Prof. Michael G. Dyer      HomePage

CS 263B - Connectionist Natural Language Processing

Addresses the issue of how Mind might reside on Brain; that is, how high-level cognitive functions, especially those required for natural language processing (NLP), might be implemented via artifical neural network (ANN) architectures. Issues include: implementing rules and dynamic bindings via ANNs, localist vs distributed representations; use of: PDP networks, recurrent neural networks, self-organizing maps, recursive autoassociative memories, tensor networks, spiking neurons, and katamic memories (in which dendrities serve as temporal delay lines).

Prerequistes: graduate standing in CS or consent of instructor (cs263A is recommended but not required).


1. Organization/Overview of Course and Review - Review of Classic and Probabilistic Approaches to NLP and Review of Basics of Neural Networks.

2. Early Connectionist NLP & Paradigms - Localist vs. PDP Approaches; Early localist/PDP networks and Critiques of Early Models.

3. Distributed Connectionist Production System - Implementing rules/variables via coarse coding, pull-out networks, and rule/symbol spaces.

4. Localist/Structured Connectionist Networks - Planning, Reasoning, Memory Organization and Retrieval, Disambiguation; Dynamic Variable Bindings: via Signatures, via Temporal Synchrony, via Relative Position Encoding.

5. Networks for Sentence Comprehension, Story Understanding and Q/A - Simple Recurrent Networks (SRNs); Parallel Constraint Satisfaction for Sentence Comprehension; Script-Based Story Understanding and Question Answering via SRNs and Self-Organizing Maps; Word Learning via Moving Target Learning (e.g., FGREP).

6. Recursive Distributed Representations - Recursive Autoassociative Memories (RAAMs), Holistic Inferences, Parsing Embedded NL Structures, Learning Distributed Semantic Representations (DSRs), Goal/Plan Analysis via DSRs.

7. Grounding Language Acquisition in Perception and Motion - Acquisition of Lexical Semantics for Spatial Terms, Association of Word Sequences with Perceptual/Motion Ssequences for Language Acquisition.

8. Selected Topics - E.g., Tensor Networks for Variable Bindings and Conceptual Representation; Approximating Semantics of Logic with Recurrent Networks.

9. Selected Topics - E.g., Extracting/Inserting Symbolic Knowledge from/to Connectionist Networks, Complexity of Learning with ANNs, Parallel Distributed Semantics Networks via Propagation Filters.

10. Selected Topics - E.g., Other Spiking ANN Models, Support Vector Machines, Parsing Noun Groups with Prepositional Phrases.

Offered: Usually every other year, in the spring quarter.

Hours per week: Lectures, 4 units (meets twice weekly)

Grading: Based mainly on course project and paper. Each student also gives a short presentation on both his/her "favorite paper" and on project status.

Readings: Students read numerous articles from a variety of sources, particularly from the journal Connection Science and from books, such as:

- Bechtel, W. and Abrahamsen, A. 2nd. ed. (2002). Connectionism and the Mind, Blackwell.
- Sun, R. and Alexandre, F. (eds.). (1997). Connectionist-Symbolic Integration: From Unified to Hybrid Approaches. Lawrence Erlbaum Associates.
Sun, R. and Bookman. L. (Eds.) (1995). Computational Architectures Integrating Neural and Symbolic Processes. Kluwer Academic.
- Levine, D. S. and Aparicio IV, M. (eds.) (1994). Neural Networks for Knowledge Representation and Inference. Lawrence Erlbaum Associates.
- Reilly, R. and Sharkey, N.(Eds.) (1992). Connectionist Approaches to Natural Language Processing. Lawrence Erlbaum Associated.
- Barnden, J. and Pollack, J. (Eds.) (1991). High-Level Connectionist Models.
- Pinker, S. and Mehler, J.(Eds.) (1990). Connections and Symbols, Bradford/MIT Press.
- Rumelhart, D. and McClelland, J. (1986). Parallel Distributed Processing (Vols. 1 , 2), Bradford/MIT Press.