The biological evolution of information processing

in the evolut= ion of language & learning,

and

my search for= the missing link between body and mind

Section 4:

A simplified software and data model for vertebrates:

an inner lang= uage

to control action and to integrate = vision

Rainer von Königslöw, Ph.D.

Abstract=

I speculate that there is a missing link, something that connects physical activity to mental activity. Furthermore, I speculate that this missing link is related to the biological evolution of language and learning. I investigate questions that are u= sually addressed in the field of neuroscience with empirical investigations. I propose a paradigm that investig= ates these questions from the perspective of information processing, and thus al= so fits into the field of artificial intelligence. I propose a design that uses an ‘inner language’ to control action sequences and to integrate visual perception into action. I investigate how this ‘inner language’ facilitates and enhances learning. The investigation demonstrates and validates the feasibility and benefits of the ‘inner language’ design with working prototypes.

TOC

Introduction: summary of previous sections. 4=

Chapter 1: Using an ‘inner language’ for the instructions to control action.. 5=

Chapter 2: Adding vision to language-= based action.. = 7=

Chapter 3: Integrating the output of = vision into the instructions for action sequences 7=

Chapter 4: Geometry and time in action sequences. <= /span>12=

Chapter 5: The functionality of the ‘inner language’ 14=

Expanded TOC<= o:p>

Introduction: summary of previous sections. 4=

Chapter 1: Using an ‘inner language’ for the instructions to control action.. 5=

Topic 1: the representation of instructions for the control processors. 5=

Topic 2: Ballet choreography as mod= el for instructions. 5=

Topic 3: using an ‘inner̵= 7; choreography as programming language to control physical action from inside= the body. <= /span>6=

Chapter 2: Adding vision to language-= based action.. = 7=

Topic 1: integrating perception to improve the action.. 7=

Topic 2: information processing for visual perception.. 7=

Chapter 3: Integrating the output of = vision into the instructions for action sequences 7=

Topic 1: the relationship between perception and action.. 7=

Topic 2: using vision to control mimicking action.. 8=

Topic 3: translating the informatio= n from vision to control mimicking action.. 10=

Topic 4: making the output of vision usable for inclusion into choreographic instructions 11=

Topic 5: the interface of visual perception and action.. 11=

Topic 6: investigating how the outp= ut of visual perception might help to control action 12=

Topic 7: the range of visually-cont= rolled mimicry across vertebrates. 12=

Chapter 4: Geometry and time in action sequences. <= /span>12=

Topic 1: “All the world’= ;s a stage” -- extending choreographic instructions to detailed specificat= ion of movement 13=

Topic 2: working with geometry: sca= ling, rotating, and translating vector images to allow comparisons and recognitio= n.. 14=

Topic 3: the management of time for choreography and for detailed specification of movement 14=

Chapter 5: The functionality of the &= #8216;inner language’ 14=

Topic 1: if … then … conditions in instructions for integrating vision.. 14=

Topic 2: the ‘inner’ la= nguage as a programming language. 15=

Topic 3: the functionality of the ‘inner’ programming language. 15=

Topic 4: nesting, the embedding of clauses within clauses. 15=

Topic 5: reuse of program code R= 11; function calls and subroutines. 15=

Topic 6: use of geometric informati= on: directions. 16=

Topic 7: use of geometric informati= on: shapes. <= /span>16=

Introduction: = summary of previous sections

Chapter 1: Usi= ng an ‘inner language’ for the instructions to control action

Topic 1:&nbs= p; the representation of instructions for the control processors
=

= Instructions for the control of action sequences might be like the different types of computer languages, from machine code through assembler, through C, to LISP, Prolog, and other high generation languages. Design of higher-level languages is associated with the design of the interpreters and compilers that translate expressions in the higher language ultimately into machine code. I will introduce the concept of action-control languages from a human perspective, and then show how it mig= ht generalize across most vertebrates.

=

Topic 2:&nbs= p; Ballet choreography as model for instructions

= I first encountered ballet in my undergraduate years, because I needed to meet a Ph= ys. Ed. requirement. Being young = and full of hormones, I opted for modern dance. Being the only guy with a group of= six beautiful girls in black leotards definitely appealed to me. Learning just the basics, enough to manage in a dance performance, was a relatively small price to pay, since I= had previously done gymnastics.

= I stumbled into ballet choreography again in the process of doing a project in animati= on. Ballet choreography uses a special= ized language for precisely specifying poses and motion for ballet dancers. Pieces might be as long as 90 minu= tes or 120 minutes. The most precise= type of ballet choreography specifies the position of the limbs and the transiti= ons from one such pose to another. I expect only ballet notation would allow someone to mimic John Cleese Minist= ry of Funny Walks (Monty Python) without first watching the performance. Furthermore we all have several st= yles of walking that we can use to convey attitudes. We can be aggressive, hesitant, af= raid, and flirtatious and convey it through our style of walking.

= Choreographic notation is probably the most precise notation relative to action that is public and shared. Since it c= an be learned, it is the furthest that mental and symbolic expressions go toward physical action. Most language specifications of action are much more vague and ambiguous, such as in stage instructions or in coaching for tennis, golf, and other sporting activities= . When used in computer animation, c= horeographic notation can produce very natural-looking motion.

= There are three components to basic choreographic commands: the action to be performed, the positioning of the action relative to the stage, and the timing of the acti= on relative to the musical score. (There may be further markings for conveying expressions, gestures, etc.) The whole idea of choreography is that it can guide different dancers to go through the same motions at predetermined locations on the stage and at the same time relati= ve to the music.

= The instructions for complex motions such as walking to a chair and sitting in = it, dancing, and jumping, are broken down into sequences of instructions for simpler movements, such as taking a stride (for walking). The simple movements can then be f= urther reduced into separate but synchronized motions for individual limbs.

Topic 3:&nbs= p; using an ‘inner’ choreography as programming language to control physical action from inside the body

I hypothesize that most vertebrates use an ‘inner’ equivalent of ballet choreography to control their actions. I think that they use precise choreography instructions for internal information processing.

= A 90 minute ballet like the Nutcracker Suite needs a lot of choreography instruc= tions that have to be remembered. T= here is a limit on memory, so we might make a rough guess that 270 instructions = are a reasonable trade-off between memory and precision. On overage each of these instructi= ons would cover about 30 seconds. It is likely that some of the choreography would be at higher resolution, where single steps in the sequence are less than 30 seconds long. Most tasks for animals and humans = are shorter than 90 minutes, so I hypothesize that they are represented by memorized sequences of 30 to 300 instructions that are stored and retrieved from memory. I also hypothesi= ze that these instruction sequences are interpreted and adapted to fit the specific circumstances. Typic= ally this adaptation involves perception.

Chapter 2: Add= ing vision to language-based action

Topic 1:&nbs= p; integrating perception to improve the action

= Process control uses sensors to monitor the process, i.e. to provide information ab= out the current state of affairs to the control computer. The sensors are polled at regular, frequent time intervals, typically in milliseconds. I see perception as the biological equivalent by providing sensory information to control action.

= Perception by itself, without leading to improved action, does not appear to have any evolutionary advantage, so I focus on the integration of information from p= erception with choreographic information in order to control action. Even though the video image may en= ter at rates of 30 frames per second, I assume rhat information integration happen= s at a slower pace, i.e. at a 3 second to 3 minute time interval.

Topic 2:&nbs= p; information processing for visual perception

= I simplify by assuming that the same approach to information processing should work for visual perception as for muscles. &nb= sp; The information originates at the retina from the rods and cones. At the retina the information is pixilated, dots in the image generated by rods and cones. I assume that the neural firing pat= tern would be sent synchronously every 30 msec (about 30 per second), the same r= ate as for muscle control. 2 such firings in sequence would correspond roughly to the flicker frequency.

= There are a lot of neuroscience results about feature extraction and other visual processing. My model is very simplistic but it should suffice for this analysis.

Chapter 3: Int= egrating the output of vision into the instructions for action sequences<= /h1>
=

Topic 1:&nbs= p; the relationship between perception and action

= In my Ph.D. thesis (1974) I modeled perception as being intentional, i.e. as a ki= nd of action initiated from the inside as the result of an instruction, rather than as a passive inflow of information.&n= bsp; I now see most sensory input as requiring action to make it happen.<= span style=3D'mso-spacerun:yes'> For vision, the action involves positioning the head and the eyes, as well as focusing. The return information flow is the visual information flowing over the optic nerve.

= There are two types of control strategies in process control. Feedback senses the outcome of an = action and adjusts the action accordingly. This strategy is also known as reaction and crisis management. The adjustment is made after the fact. Bumping your leg and st= opping is one example. Groping to fi= nd your glass of water at night is another.&n= bsp;

= Feed forward takes information from earlier in the process to improve action lat= er in the process. This strategy= uses prediction and planning. The adjustment is made before the fact. For example, we first see a chair and then walk over to it to sit do= wn. Visual perception illustrates both strategies in cooperation. We= look for the chair by swiveling the head and focusing the eyes until we recognize the appropriate shape. Moving= the head until … illustrates feedback.&n= bsp; Seeing the chair, then turning and walking – illustrates feed forward, assuming we predict how much to turn and how far to walk until loo= king for the chair again.

Topic 2:&nbs= p; using vision to control mimicking action

= The second set of examples comes from mimicking action. A typical example arises in a fitn= ess, yoga, or ballet class, where the student wants to imitate the actions of the instructor. The instructor= 217;s pose or motion has to be seen and analyzed, and compared to one’s own pose or motion. Differences detected in the comparison have to be translated into action to reduce the differences.

= To make it work, we have to use a representation, both of the individual to be mimicked, and of the self. &nbs= p; We have to compare the figures, detect the differences, and generate control instructions for actions to minimize the differences.

<= span style=3D'font-size:14.0pt;font-family:Garamond'>

= I go to fitness classes in the YMCA, where I follow instructors. Most of them lead by example, with= out using words. So while I am tr= ying to work up a sweat I can speculate on how the information might flow from my eyes to my arms and legs. Unfortunately I have no access to the insides of my mind or my brain= . So I resort to speculating on how = I can make a computer simulate this vision-based action copying. One way to make this work is to us= e a stick-figure representation of the self and of the instructor in something = like a scalable vector graphics representation.= By rotating and scaling the two stick-figure representations, I can superimpose them and match up joints and limbs. For instance, if I match up the up= per arms and the elbow joints then I can ‘read off’ (calculate) the difference in joint angle for the lower arm. I can then insert this difference = in angle in an appropriate action command that should reduce the difference. Initially, with a new exercise or = a new instructor, it may take me several steps or adjustments to get it approxima= tely right. After a while I can ge= t it the first time. After that, o= nce I am really used to the routine, I don’t even have to look.<= /span>

= Mimicry is an example of apprenticeship learning that seems to happen for many species. One such example is imprinting by ducklings, where they recognize the mother hen and waddle aft= er her.

Topic 3:&nbs= p; translating the information from vision to control mimicking action<= /a>

= Recognizing an object involves comparisons, so we need a representation as output of vi= sion that supports the geometric transformations that have to be made to correct= for perspective. The transformed representation of the object may have to be scaled, rotated, and translated= so that it can be superimposed and compared to the same kind of object. If the differences are few and sma= ll enough, it can be recognized as the same or at least as the same type.=

I speculate that pixelated or raster-scan visual images from the retina are processed to extract a vector representation, with lines and angles. A vector graphic representation is somewhat like drawing the object schematically, with lines and curves. Objects can be represented in 3D l= ike in CAD drawings for engineering and architecture.

I hypothesize that we can generate and internal vector representation of the = instructor, like a stick figure. I also hypothesize that I have a representation of myself, also as stick figure. As first step, I can rotate and translate the two stick figures so that they are superimposed. If I match them up so both bodies = have the same orientation and that the hip joints coincide, I can then compare t= he relative angle of the upper leg. I can then issue commands to my hip joint to bring it in line with the instructor. I can now superim= pose the knees and find out how much I have to rotate the knee to bring them in line.

= I would like to start with some examples. I already mentioned recognizing and locating a chair so that I can walk to it= and sit down. Recognizing an obje= ct such as a chair involves comparisons, so we need a representation that can = be produced as output of vision, that can be stored and retrieved from memory,= and that produces information that is suitable as input for action. Another similar situation arises wh= en a cat wants to chase a mouse. F= irst the cat must recognize the mouse as potential prey. Then the cat has to decide in what direction to run.

Topic 4:&nbs= p; making the output of vision usable for inclusion into choreographic instructions

A blackboard would be wonderful. Stage directions and poses can be conveyed visually with simple line drawings and stick figures. Architects and engineers use CAD drawings to convey designs and guide construction. Blackboards are= also used by football coaches to sketch strategies with line drawings. We could draw a remembered chair n= ext to a perceived chair and make comparisons.&nb= sp; Unfortunately, neuroscientists have so far been unable to find anyth= ing like it in the brain, even though the concept was popular in artificial intelligence circles in the 70’s.

= The visual information has to be encoded in zeros and ones, so that it could potentially be similar to neural encodings. Drawings use lines, curves, and di= agrams with horizontal and vertical axes. Lines can intersect with angles.&nb= sp; Computer representations of drawings also face the problem of encodi= ng drawings into data (zeros and ones), and do so in a variety of ways. I have been experimenting with a n= ew graphic language that is becoming a standard on the internet, called SVG for Scalable Vector Graphics, to try to make it work. (Before that I experimented with O= penGL, but thought that representation was too rich and powerful, i.e. overkill.)<= o:p>

Topic 5: the interface of visual perception a= nd action

= At the interface of visual perception and action, it makes sense to have a fairly standardized representation. = It seems unlikely that the internal representation changes significantly as we move the head and thus the viewpoint of the eyes. It makes sense to have the remembe= red image similar to the perceived image. It seems unlikely that the internal representation of objects changes significantly as we come closer. Artists learn about proportions and about colours, e.g. the different colours of sn= ow that are rarely pure white.

= We come back to our concepts of choreography and stage directions. It makes sense to visualize or inte= rnally represent the positions of our bodies, limbs, and head relative to the stag= e, the floor, or the direction of gravity. It makes sense to visualize or inte= rnally represent the positions of other actors or of objects such as chairs relati= ve to that same stage and in comparable size units.

= We might assume that the representation does not change significantly as we move aro= und the stage, or as the other actors or objects move or change positions. We therefore need to cope with thr= ee dimensions rather than the two in the retinal image or in SVG.

=

Topic 6:&nbs= p; investigating how the output of visual perception might help to cont= rol action

=

= To further investigate this hypothesis, I need to simulate some examples. At present I am still working on t= he stick figures, the scalar vector comparisons, and the automatic generation of act= ion commands. <= /p>
=

= My earlier example was seeing a chair, walking toward it, and sitting down on it. I imagine that this invol= ves a comparison and difference reduction. In this case the stick figures are myself and the chair on a stage.<= span style=3D'mso-spacerun:yes'>

=

= As first step, I can rotate so that I am facing the chair. I can issue commands to the hip jo= int and the knee to slightly lift one leg and rotate on the other.

=

= As second step I can walk toward the chair.&n= bsp; The walking instructions are high level, and have to be interpreted = into detailed instruction for the right leg and the left leg. And for both of these there have t= o be more specific instructions to the hip and to the knee. At some stage during this walk I am likely to have another look at the chair and to correct my action plan.

=

= As a third step, I am likely to stop in front of the chair, possibly circle it partway if it is facing in a direction different than the line of travel. Then I will lower myself into the = chair until I am sitting.

=

Topic 7:&nbs= p; the range of visually-controlled mimicry across vertebrates

=

= Fish swim in schools in quite an orderly fashion, with remarkably synchronized t= urns. Birds fly in flocks, and ducks imprint. In most if not all of these cases it seems likely that vision plays a central role in coordinating the actions.

=

=

Chapter 4: Geo= metry and time in action sequences

=

Topic 1:&nbs= p; “All the world’s a stage” -- extending choreograph= ic instructions to detailed specification of movement

=

= Reference to the stage and to gravity ‘down’ makes sense to anchor percep= tion.

= Explicit written or spoken choreographic instructions are typically expressed relati= ve to the stage. However, these references must be interpreted into joint rotations relative to other body parts before they can be turned into instructions for joint rotations throu= gh relative muscle tensions.

=

= For example, an instruction might tell us to hold the lower left arm horizontal= ly relative to the stage. This instruction must be broken apart into separate instructions for each muscle involved. Joint rotations are= in up to three dimensions managed by complex sets of muscles. We can simplify the problem slight= ly by focusing on the joint rotation, since the muscle tension also depends on the load on the limb and thus on the muscles.&= nbsp; A geometric description relative to the stage has to be translated i= nto a geometric description relative to the bone the joint is attached to.=

=

= I assume that the angle of the hips or the shoulders relative to the stage is ‘known’, possibly through proprioceptive information such as balance. The upper legs are relative to the hip, the lower legs relative to the upper leg, and the feet= relative to the lower legs. Similarly = the upper arms are relative to the shoulder, the lower arms relative to the upp= er arm, and the hands relative to the lower arms. The neck is relative to the should= er, and the head is relative to the neck. With this set of relationships, I have greatly simplified the skelet= on, but it should suffice for my exploration of action.

=

= Instructions can be ambiguous. For our exa= mple above, to hold the lower left arm horizontally relative to the stage, the shoulder and elbow bends are different depending on whether the upper arm is parallel to the stage or is vertical. As second example, how I hold my hand parallel to the ground depends= on whether I am standing upright or lying on my stomach. It also depends on whether my arm = is hanging down, held straight in front of me, or is pointing up. If I am standing up with my arm do= wn, I have to bend my wrist up. If I am standing up with my arm straight in front= of me and parallel to the ground, then the wrist can be straight. If I am stan= ding up with my arm up, I have to bend my wrist down to get my hand approximately parallel to the ground.

=

Topic 2:&nbs= p; working with geometry: scaling, rotating, and translating vector ima= ges to allow comparisons and recognition

=

= The image captured by the eyes changes every time the head moves. We can also move the eyes and focus. The visual image, ther= efore is the outcome of an action that affects the viewpoint or perspective of wh= at is seen. Interpreting the ima= ge requires geometry to figure out what we are seeing and its relationship to where we are. I am sure we ha= ve all had the experience of lying on the floor and seeing someone’s boots up close and huge, while the head seems far away and barely visible.

=

= For figuring out what action to take, such as sitting up or standing up, we use geometry to figure out where the person is, relative to us. Even for recognizing the person we= need some geometric transformations, since we have probably never looked up the nostrils of that person before. Somehow we can ‘correct’ the visual information to allow= for the position we are in and the perspective from which we are looking. =

=

Topic 3:&nbs= p; the management of time for choreography and for detailed specificati= on of movement

=

= Motions have to be synchronized quite precisely to avoid stumbling or other mishaps. There is a delay fro= m the initial high-level choreographic command and the start of the flow of low-l= evel instructions to the muscles. = This is a ‘computing’ delay, because the high-level instruction has = to be interpreted and expanded into the detailed micro-instructions. There is an even greater delay fro= m the initial high-level choreographic command to look for the chair. I first have to orient the head and= focus the eyes. I then have to inte= rpret the visual image to recognize the chair.&n= bsp; If I don’t recognize the chair I have to go back to turn the h= ead further. Only after that can = I plan and execute the action.

=

=

Chapter 5: The= functionality of the ‘inner language’

=

Topic 1:&nbs= p; if … then … conditions in instructions for integrating vision

=

= To make action more successful from an evolutionary perspective, we need conditionality. A simple vers= ion is ‘if you see this then follow action sequence 1, otherwise follow acti= on sequence 2’. In essence= we are enriching the ‘inner’ language with connectives. From another perspective, we might= see this as a progression where we evolve toward a better internal programming language. Presumably this hap= pened in stages. =

=

Topic 2:&nbs= p; the ‘inner’ language as a programming language

=

= I see ballet choreography as the beginnings of a programming language for action sequences. A ballet sequence = is meant to be deterministic, i.e. it is not meant to give the dancer choices = on whether to go to this side of the stage or the other. For hunting or food gathering we n= eed choices. We don’t want = to eat poison mushrooms, and we don’t want to attack a predator that might e= at us. So it is logical that conditions entered early in the development of the ‘inner’ language.

=

= Similar conditionality might allow feelings to select action sequences. For example, we might have ‘= If I am hungry then …’, and ‘If I am tired then …’= .

=

Topic 3:&nbs= p; the functionality of the ‘inner’ programming language

=

= Since there are different types of mushrooms that are good to eat, it makes sense= to find an ‘or’ connective early in the development. We should also find ‘andR= 17;, and ‘not’, since we might want to eat apples that are big and n= ot green.

=

= At a later stage in the evolution of the ‘inner’ language we might f= ind the programming capability to write, edit, and execute programs. Such capabilities might underlie innovative play and the creation of strategies for hunting that may occur in some cats, monkeys, etc.

=

Topic 4:&nbs= p; nesting, the embedding of clauses within clauses

=

= We might want to eat a mushroom only if we can see one. We would only look if we are hungr= y and not too tired. We would consi= der any of that of we are not afraid of some predator and fleeing or hiding.

=

Topic 5:&nbs= p; reuse of program code – function calls and subroutines
=

= Looking for a mushroom should be similar to looking for potatoes or fallen apples.<= span style=3D'mso-spacerun:yes'> We should therefore be able to use= much of the same functionality such as looking around on the ground with a search pattern, and comparing the perceived shape with the target shape.

=

Topic 6:&nbs= p; use of geometric information: directions

=

= Much of the action involves directions. If we see a desirable mushroom we want to walk toward it. We then want to reach out and grab= it. We definitely do not want to walk = in the wrong direction or to reach out and miss.&= nbsp; We cannot just use eyesight to correct our movements since we might = have to watch out for roots to avoid stumbling.

=

Topic 7:&nbs= p; use of geometric information: shapes

=

= The information from vision is used in comparisons to recognize mushrooms and apples. The outline shape may= be used, or an inferred 3D shape, or even surface textures and colours.

=

Chapter 3: Int= egrating the output of vision into the instructions for action sequences<= /h1> =

Chapter 3: Int= egrating the output of vision into the instructions for action sequences<= /h1>
=