The scientific objectives of the long-term research programme

The objective of the research programme is to investigate a new paradigm for language comprehension. The traditional approach is based on the assumption that language is for communication. It is also assumed that there is a common grammar and a shared semantic network among members of a language community, as demonstrated by professional qualifying exams. Computational models have focused on simulating language behaviour, primarily on questions and answers (description rather than instruction, persuasion, or entertainment).

This research paradigm starts with the assumption that language is a brain-based facility for faster and wider-scope learning for individuals, through self-programming. Communication is seen as secondary. A further description of this paradigm is attached: "Body, brain, language".

The research programme is dedicated to exploring the feasibility and limits of this theoretical framework with a number of computational prototypes.

The research will be done in phases. The first phase seeks to build a simulation framework with simplified components that address the major components of the theory, to support the overall approach. The simulation will be extended in future phases and will seek to provide better support for the theory. At the same time, there will be an attempt to spin off practical applications of this theory.

At this point, the system is for prototyping and testing concepts, not for embedding in games. At most it may be used as front-end to a proper animation engine to illustrate how it might work with a gaming engine. That may be enough to elicit future contracts.

Scientific objectives:

To develop a new theory of language comprehension that is based on a reasonable thought-brain-body model which incorporates the universality of genes and grammar while respecting the differentiation of languages and cultures.

To develop a methodology of computational experimentation for verifying the new theory

The revised model sees the brain as equivalent to a distributed process control system, with programmable logic controllers for brain-body functions, and a central system for the thought-brain functions.

Hypothesis 1: the PLC equivalents in the brain interpret 'subconscious' instructions, and the central unit interprets thought

chunking PLCs should consider both brain wiring and decomposable instruction sets. For simplicity and convenience we have started with a single PLC model for major body movements such as walking and sitting (controlling the major skeletal units).

Hypothesis 2: there are two types of such instruction. The PLC programming represents basic, universal skills, i.e., not language or culture dependent. These skills cover both action and perception. The central unit instructions are encoded in language form, and usually accessible to conscious introspection, and reasoning.

PLCs are relatively independent, work in parallel, and probably have a 24/7 duty cycle

The main central system is primarily serial, with a 16/7 duty cycle, the wake / thinking cycle. (There is one or more secondary central system with an 8 / 7 duty cycle for the sleep period. This system will not be addressed at this time, but may be important for 'recompiling' both central system and PLC instruction sets)

Hypothesis 3: the central unit programs the PLC instructions, and the central unit programs itself.

Program self-modifications possibly use the sleep/wake cycle for major revisions and 'rebooting'

The central unit sends instruction sets to the PLCs to execute high level actions

The PLC translates and expands these instructions to send detailed binary instructions over neural wiring to muscles, with more detailed and frequent timing than the thought rhythm.

Hypothesis 4: there are at multiple computational layers in working with natural language encoded instructions originating in conscious thought and ending with physical (muscle and limb) action. These layers represent the mind - body interactions, by translating language instructions into neural instructions, and the reverse. (Similar considerations apply to images, etc., but the focus here is on language). The computational experimentation seeks to discover reasonable translation algorithms, focusing on time and scope resolution.

These layers are grouped, for simulation purposes, so in effect we are modeling 2 layers

Group 1: the physical system, the PLC, wiring, actuators, and sensor systems

the body, with a jointed skeleton, muscles and perceptors

the hardware PLC component of the brain, with neural wiring to the muscles and perceptors

'brainware', PLC oriented machine-code type of stored instruction not open to introspection. This is presumably a universal brain-representation that is independent of culture and upbringing, but that may fit into Chomsky universal structural components.

Timing for information flow through these components is fast, in neural firing sequence. There is some delay, since instructions currently being executed by muscles had to be processed in the brain and then propagated though the neural wiring.

There is little or no memory, since instructions flow through the system, and muscles can fatigue but do not remember complex sequences.

There is some but little information flow that bypasses the central control system. An example of such information bypass is the reflexive catching of a ball without thought or attention - a kind of automatic or interrupt-driven eye-hand coordination.

Group 2: the central control system, where a single plan is evaluated and executed

'thoughtware', natural language encoded sets of stored instructions that are generally open to introspection and conscious analysis

This might include conscious and subconscious knowledge and skills, generally culture and context dependent.

It includes thought and reasoning as well as learning (in school and other contexts). It may also include recalling and acting on memorized scripts, such as for actors.

The timing for this is much slower, and generally serial in a single thread.

Interaction with the various group 1 systems seems to be through a content and speed limited network, often described under the heading of attention.

Hypothesis 5: there are 4 interfaces that simulate the information flow between the layers

In complement to the 2 groups discussed under hypothesis 4, we are simulating 3 interfaces

the external world and the body

The 'hardware' component is likely limb-specific, i.e., fairly localized. A higher level of 'programming' is required to synchronize the movement for walking.

The body is modeled as a tree of connected cylinders, angled relative to one another (the externally visible, covered skeleton)

movement is through changes in limb positions of the skeleton

body-time and world-time is synchronized at 30 frames per second

the language instructions and the PLC programming (brain-body)

low level instructions to change relative limb positions (joint angles)

based on a desired goal position for each limb as part of the next continuous move (e.g., bring leg forward to ... as part of single step in walking)

timing is based on an internal rhythm that allows some synchronization between the movements of different limbs, i.e., left-leg and right-leg

timing of movements has to be synchronized, such as leg and arm swings for walking.

'thoughtware', knowledge, and scripts

at the highest level expressive and purposive commands may be used, such as 'walk bent over' or 'walk to the chair'. These can be consciously translated into high level commands, such as "walk forward 5 steps", and "let the arms hang".

Hypothesis 6: 'learning' refers to distinct processes with very different properties in each of the 5 layers

At this stage, self-programming and learning is not yet addressed, except at a trivial level, i.e., memorizing a new script

Hypothesis 7: The overall model should be compatible with the overall theory of evolution, since other, especially mammalian species can have limited prototypes of these mechanism to support both learning and planning activities without overt consciousness or language use. In other words, in these species pre-language might be used for self-programming but not for communication.

Technological objectives:

An actor acting in a play with a script and stage directions is a simple and explicit illustration.

To design and develop the technology which supports computational experimentation with this layered architecture in a virtual reality.

This project seeks to mature the brain-body-world base for the computational model, before focusing more on mind-thought-language.

The architecture must support parallel, somewhat independent processing.

For example, a person might walk down the street while reading about, and imagining being in a fantasy world.

Computationally, the walking must be controlled by one process, while the reading and imagination are involving separate and independent processes.

Parallel timing representations should preserve sequences and task dependencies (first do this then do that), yet allow for subjective time, then map into neural time and real-world time.

When transferred from thinking to action, hypothetical steps and timing must be converted to sequences.

Action components must be synchronized, e.g., continuing walking while reaching out with a hand.

The brain-body timing should be more reasonable, i.e., a few frames from program invocation to body response

At the same time, the brain-body-world model should be improved to be more reasonable in terms of what is known.

The world representation must form the basis for visual (eye) input as well as for allowing multiple views from multiple cameras.

The data representations especially at the interfaces should be reasonable and ideally support verification

The brain-body-world model should also be compatible with the theory of evolution, i.e., the basic scripted action to neural action should not contradict what is known about mammals and other animals.

Attention mechanisms should be explored, i.e., the transfer of information from sensory input and thought control to the body-action module. Initial evidence linking autism to disturbances in brain rythms seem to support this approach.

If possible, interruptability should be explored, e.g., body-sensory feedback versus a 'Stop' verbal command

One question is whether one can simulate reasonable brain-body functionality without drowning in overwhelming complexity, but still use it as a base for exploring language functionality and a self-programming model of learning.

At a lower level of abstraction, is it possible to find a reasonable data representation for the world-data representation. It has to deal not only with physical movement produced by the actor but also with the perception of the physical movement of other actors on the stage. Later it has to be extended to speech, etc.

Is it possible to model timing, so action of arms and legs can be coordinated and yet be somewhat independent and programmed separately (by a preceding thought)

Eventually well-known synchronization problems should emerge, such as the difficulty of tapping with one hand, making a circle with the other hand, and doing both at the same time at different speeds.

Is it possible to come up with reasonable data representations to represent habitual action, skill, and planned and intended action

Aug. 2004 - March/April 2005 --

The focus was on exploring the distributed process control model, exploring timing and data representation issues

world-body: connected cylinders in frame - in db, generatable by the body but visible in the world, perceivable from different angles

Presently uses OpenGL - but it would be better to use a physics-style matrix representation with 3D, but expandable to force and mass, and then to use SVG to isolate viewing.

neural brain functions: changes in joint angles over time increments (represents move toward goal states)

physical (world) action time is represented as frames, in fixed-rate frame per sec.

brainware: low-level 'move bone' commands, where the timing is represented in beats, a subjective measure related to body rythms

low-level commands are interpreted into limb and joint goal states, by position and time

subjective and objective time are not yet independent

transition to goal state is linear (over rotation and time)

body-brain-PLC functions: medium-level commands, interruptible in chunks, can simulate delay

synchronization is not working at this level, simulated by sorting over low-level command timing

thoughtware: high level 'walk', 'sit' commands (conscious action)

April 2005 - Aug. 2005 --

Experimentation with medium-level and low-level script representation to allow asynchronously received script fragments to be resynchronized before being translated into body action. This experimentation was only partially successful. A better marking scheme needs to be devised for the high-level script to identify synchronization opportunities without limiting flexibility.

An example is swinging your arms while walking. You can walk without swinging arms. You can start swinging half-way through a walk sequence. You are unlikely to continue swinging after you stopped walking.

You can decide to start swinging you arms, e.g., for picking up the pace - so the instruction (script) is sent separately from walking instructions, or the walking style instruction is changed.

A couple of approaches have been tried, but none very satisfactory so far.

For now, synchronization is simulated by sorting over low-level command timing

Aug. 2005 - Dec. 2005 --

A redesign of the scripting language and scripting elements, including high, medium, and low-level script representation. Planning and synchronization at this level is anticipated, but for now only violations of time-ordering and time-continuity are identified.

The redesign is not really successful, and points out the need for a clearer theoretical analysis of scripts, script expansion (macros), and script interpretation into basic elements.

The theoretical analysis brought out similarities in scripting design for Web-based process control applications, where real-time management of processes is involved, and where communication is unreliable (attention - Web traffic).

For the Web, multiple clients may gather and submit data, multiple servers may be involved in storing and interpreting the data, and multiple clients may provide supervisory interfaces.

Overlapping parallelism as discovered for brain-body controls may be very useful and may be tested in this setting.

Results and plans for the future

While not yet completely explored, the approach seems to work reasonably well for modeling physiological knowledge.

In general, no conflicts with the theory of evolution have been encountered, so that this approach may be yet another way of addressing the mind-body debate.

Medium-level scripting used for PLC programming chunks might well correspond to Chomsky grammatical structures as well as to known brain structures

Timing must be modeled more reasonably

Attention is modelled as a limited-bandwidth network connection that uses script-like language fragments as communication protocol.

Interruptability is still a problem that must be addressed

At present the simulation runs on 5 networked computers with 5 simultaneously running, independent but interacting applications. The simulation therefore demonstrates a fairly high level of parallelism, even though the theory calls for even more.

Hypothesis 1: the human brain supports the interpretation of instructions stored in memory

Hypothesis 2: there are two types of such instruction. One type is encoded in language form, and usually accessible to conscious introspection, and reasoning. The other is universal, i.e., the same for all humans (reflecting the same genetic capacity). This type appears to be subconscious but may be reflected in feeling states, etc.

Hypothesis 3: the brain self-programs these instructions, and/or the instructions are self-modifying.

Hypothesis 4: there are at least 5 computational layers in working with natural language encoded instructions in conscious thought

the external world

the body, with a jointed skeleton, muscles and perceptors

the brain, with its various functions and centres

'brainware', a machine-code type of stored instruction not open to introspection, a universal brain-representation that is independent of culture and upbringing

'thoughtware', natural language encoded sets of stored instructions that are generally open to introspection and conscious analysis

Hypothesis 5: there are 4 interfaces that constrain the information flow, both content and time

the external world and the body

current limb positions of the skeleton is part of the action interface

the body and the brain

nerve instructions to the muscles to change relative limb positions (joint angles)

the brain and 'brainware'

desired goal position for each limb as part of the next continuous move (e.g., bring leg forward to ... as part of single step in walking)

'brainware' and 'thoughtware'

high level command, such as "walk forward 5 steps"

Hypothesis 6: 'learning' refers to distinct processes with very different properties in each of the 5 layers

the external world: e.g., evolution

the body: e.g., selective muscle strengthening

the brain: new or changed neural connections

'brainware': new habits, possibly learning a new language

'thoughtware': memorizing a play, studying in school

Technological objectives:

Development of a game-action environment that supports prototyping and experimenting with the multi-layered language-comprehension, learning, shared reality paradigm hypothesized in the theory being developed.

To design and develop the technology which supports experimenting with this layered architecture in a virtual reality.

Multiple actors on a stage, acting in a pre-scripted play

Stage directions represented in simplified English

Actions are represented by skeletal movements (body functions)

The translation of stage directions into actions must fit the five layered architecture, and reasonable time and memory constraints associated with each layer

There must be support for multiple actors on the same stage

The external world must support multiple viewpoints, to map into the perceptions of actors

Visual perception and recognition of other actors and obstructions must be supported, and must fit the five layered architecture, and reasonable time and memory constraints associated with each layer

Speech production and recognition must be supported, and must fit the five layered architecture, and reasonable time and memory constraints associated with each layer

Very limited learning, i.e., self-modification of the script based on speech recognition of director's changes

Very limited learning, i.e., self-modification of the script based on visual recognition of obstacles

Computational theories of language comprehension have focused on internal semantic representations with a question answer approach. There has always been a quiet assumption of a universal semantic network, essentially a shared reality, somehow parallel to the universal grammar of Noam Chomsky. The approach taken here takes the evidence of a universal grammar as indication for the structure of the brain-based instruction set processor. The natural language interpreter is then assumed to be learned individually, clearly a primary task of young children, but not one for which we have computational models.

The six key hypotheses are novel, and, if true, could have wide-ranging impact on several fields of science, not just artificial intelligence.

Artificial intelligence programmes generally simulate 'intelligent' behaviour. Generally there have been few constraints on the architecture, as long as it succeeds in the functionality. In this case we are also imposing structural constraints, where each layer of the architecture should exhibit the appropriate functionality. A new technological platform with the appropriate layering is required for this investigation.

Appropriate measures of 'goodness of fit' have to be invented, to indicate how well a given simulation provides feasibility or correctness support for the six hypotheses. This initial project seeks to provide simulation support for the layered approach using a simplified natural language comprehension approach. The learning component is not yet addressed.

Time is being explored as multiple time streams, including:

world-time

neural & muscle-time

sub-conscious time

thought-time

These are being explored as command-time and delta-frame time, but the approach is not very satisfactory since there are limits in exploring time-shifting and commits.

The brain and 'brainware' interface is still very unclear. The concept of a human universal brain to 'brainware' interface is central to the theory, but it is still very unclear how to structure it as an API so it could work computationally and be corroborated by independent evidence.

D. Description of Work in this Taxation Year

Sept. 2002 - Feb. 2003 -- First version, layered but without time or memory constraints

external world: stick figures on wire-mesh stage, single perspective

body functions: tree structure of bones with joints, joint angles

brain functions: changes in joint angles over time increments

brainware: low level 'move bone' commands

thoughtware: high level 'walk', 'sit' commands

Feb. 2003 - Aug. 2003 -- Second version, world & body time in frames, thought time in beats, add director

external world: represented by video camera with playback

external world: add director to start actor, play video

thoughtware: add script concept to separate knowledge from stage directions

Results and plans for the future

The approach works for gross motion such as walking across a stage, and sitting. Hand motion, requiring coordination of many bones, is still problematic and will be explored later, possibly with language extensions.

language comprehension for the stage direction script is implemented as fairly simple macro expansion following a simple rhythm.

Nerve to muscle motion controls are implemented as incremental joint angle changes in 3 dimensions in 30 frames per second discrete intervals.

The experimentation showed the weakness and incompleteness of the initial model for brain to body information flow which need to be addressed before going further with the thought to brain model.

The revised model will be explored in a new project, described below.

To develop a methodology of computational experimentation for verifying the new theory

Hypothesis 1: the PLC equivalents in the brain interpret 'subconscious' instructions, and the central unit interprets thought

PLCs are relatively independent, work in parallel, and probably have a 24/7 duty cycle

Hypothesis 3: the central unit programs the PLC instructions, and the central unit programs itself.

Program self-modifications possibly use the sleep/wake cycle for major revisions and 'rebooting'

The central unit sends instruction sets to the PLCs to execute high level actions

The PLC translates and expands these instructions to send detailed binary instructions over neural wiring to muscles, with more detailed and frequent timing than the thought rhythm.

Hypothesis 4: there are at least 5 computational layers in working with natural language encoded instructions in conscious thought. These layers represent the mind - body interactions, by translating language instructions into neural instructions, and the reverse. (Similar considerations apply to images, etc., but the focus here is on language). The computational experimentation seeks to discover reasonable translation algorithms, focusing on time and scope resolution.

These layers are grouped, for simulation purposes, so in effect we are modeling 2 layers

Group 1: the physical system, the PLC, wiring, actuators, and sensor systems

the body, with a jointed skeleton, muscles and perceptors

the hardware PLC component of the brain, with neural wiring to the muscles and perceptors

timing for information flow through these components is fast, in neural firing sequence. There is some delay, since instructions currently being executed by muscles had to the processed in the brain and then propagated though the neural wiring.

There is little or no memory, since instructions flow through the system, and muscles can fatigue but do not remember complex sequences.

Group 2: the central control system, where a single plan is evaluated and executed

'thoughtware', natural language encoded sets of stored instructions that are generally open to introspection and conscious analysis

This might include conscious and subconscious knowledge and skills, generally culture and context dependent.

It includes thought, reasoning, school and other learning. It may also include recalling and acting on memorized scripts such as for actors.

The timing for this is much slower, and generally serial in a single thread.

Hypothesis 5: there are 4 interfaces that represent the information flow, both content and time. This is intended as a corollary of hypothesis 4, but focusing on information representation.

In complement to the 2 groups discussed under hypothesis 4, we are simulating 3 interfaces

the external world and the body

The body is modeled as a tree of connected cylinders, angled relative to one another (the externally visible, covered skeleton)

movement is through changes in limb positions of the skeleton

body-time and world-time is synchronized at 30 frames per second

the language instructions and the PLC programming (brain-body)

low level instructions to change relative limb positions (joint angles)

based on a desired goal position for each limb as part of the next continuous move (e.g., bring leg forward to ... as part of single step in walking)

timing is based on an internal rhythm that allows some synchronization between the movements of different limbs, i.e., left-leg and right leg, as well as legs and arm swings.

The 'hardware' component is likely limb-specific, i.e., fairly localized. A higher level of 'programming' is required to synchronize the movement for walking.

'thoughtware', knowledge, and scripts

Hypothesis 6: 'learning' refers to distinct processes with very different properties in each of the 5 layers

At this stage, self-programming and learning is not yet addressed, except at a trivial level, i.e., memorizing a new script

Technological objectives:

The objectives have not changed significantly from the first prototype. Some changes in the architecture are required to reflect the change in the conceptual model, with the brain as a distributed process control system.

An actor acting in a play with a script and stage directions is a simple and explicit illustration.

To design and develop the technology which supports computational experimentation with this layered architecture in a virtual reality.

This project seeks to mature the brain-body-world base for the computational model, before focusing more on mind-thought-language.

B. Technology or Knowledge Base Level

The starting point is the results of project LBA above.

Despite an ongoing review of the literature, no reports or technology directly pertaining to this model or approach was found.

C. Scientific or technological advancement sought

The main advance sought is to change the computational model to explore and reflect the new theoretical model, i.e., to go from a general instruction processor to a distributed process control model with multiple PLCs for brain-body functions and a central system for mind-language functions.

At the same time, the brain-body-world model should be improved to be more reasonable in terms of what is known.

The brain-body timing should be more reasonable, i.e., a few frames from program invocation to body response

If possible, interruptability should be explored, e.g., body-sensory feedback versus a 'Stop' verbal command

The data representations especially at the interfaces should be reasonable and ideally support verification

Scientific uncertainty

Using a distributed process control model for the mind-body problem is novel, including the PLC model for brain localizations.

Technological uncertainty

Is it possible to model timing, so action of arms and legs can be coordinated yet be somewhat independent, and programmed by a preceding thought

Is it possible to come up with reasonable data representations to represent habitual action, skill, and planned and intended action

D. Description of Work in this Taxation Year

Aug. 2003 - March/April 2004 --

The focus was on exploring the distributed process control model, exploring timing and data representation issues

world-body: connected cylinders in frame - in db, generatable by the body but visible in the world, perceivable from different angles

body-brain-PLC functions: timing independent, interruptible in chunks, can simulate delay

brain functions: changes in joint angles over time increments (no major change)

brainware: low level 'move bone' commands (no major change)

thoughtware: high level 'walk', 'sit' commands (no major change)

Results and plans for the future

While not yet completely explored, the approach seems to work better for modeling physiological knowledge.

PLC programming chunks might well correspond to Chomsky grammatical structures as well as to known brain structures

Timing can be modeled more reasonably

E. Supporting Information

A number of papers analysing the problem and exploring the theory as well as potential approaches to measurement and verification

2 papers are attached: "Conjectures", and "DCS-PLC"

Programme Name - Language based learning

The objective of the research programme is to investigate a new paradigm for language comprehension. The traditional approach is based on the assumption that language is for communication. It is also assumed that there is a common grammar and a shared semantic network among members of a language community. Computational models have focused simulating language behaviour, primarily questions and answers.

This research paradigm starts with the assumption that language is a brain-based facility for faster and wider-scope learning for individuals. Communication is seen as secondary. A further description of this paradigm is attached: "Body, brain, language".

The research programme is dedicated to exploring the feasibility and limits of this theoretical framework with a number of computational prototypes.

The research will be done in phases. The first phase seeks to build a simulation framework with simplified components that address the major components of the theory, to support the overall approach. The simulation will be extended in future phases and seek to provide better support for the theory. At the same time there will be an attempt to spin off practical applications of this theory.

This project is applied research in artificial intelligence, specifically in language comprehension.