15th International Summer School


Cognitive Science

NBU, Sofia, June 30 - July 19, 2008



Language Processing


Jeff Elman

University of California, San Diego




Language has been one of the most fruitful domains for studying human cognition. Not only does it play a central role in human activity, making possible cultural, social, and intellectual activity that is unparalleled among the animal kingdom; but it is also a domain that is richly complex and attractive as an arena for formal modeling. Not surprisingly, the history of modern computation, cognitive theory, and language reveal tight and important interconnections.


The major focus of this course will be on language, from a connectionist perspective. We will begin with a brief historical review of the intellectual roots of modern cognitive science (from the mid-1800s to present). The bulk of the course will then a set of phenomena in psycholinguistics and language acquisition, viewed from a connectionist perspective, and concluding with a discussion of implications for linguistic theory.



Day 1: Historical roots of modern cognitive science

        Psychology in the 19th century


        Cybernetics, computation, AI

        The cognitive revolution

        Cognition revised: Connectionism, Artificial Life, Situated & Embodied Cognition, Dynamical Systems


Required Readings


A short history of AI



Von Neumann, J. (1948/1963). The general and logical theory of automata. In A.H. Taub, Ed., John von Neumann, Collected Works, Vol. 5. Oxford: Pergamon Press.



Turing, A.M. (1950). Computing Machinery and Intelligence. In Readings in Cognitive Science: A Perspective from Psychology and Artificial Intelligence, (1988). A. Collins and E. E. Smith (Eds). Kaufmann, San Mateo, CA.

        (Only available on CD)



Optional Readings


Chomsky, N. (1959). On certain formal properties of grammars. Information and Control, 2, 137-167.

Chomsky, N. (1959). A review of B. F. Skinner's Verbal Behavior. Language, 35, 26-58.




Day 2: Early connectionist language models


        The word superiority effect model

        The TRACE model of speech perception

        The problem of learning, and a solution

        Rules or networks: The past tense debate


Required Readings

Rumelhart, D.E., & McClelland, J.L. (1986). On learning the past tenses of English verbs. In J.L. McClelland and D.E. Rumelhart (Eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 2. Cambridge, MA: MIT Press. Ch. 18.


Rumelhart, D.E., Hinton, G.E., & Williams, R. (1986). Learning internal representations by error propagation. In D.E. Rumelhart and J.L. McClelland (Eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1. Cambridge, MA: MIT Press. Ch. 9.


Annotated reading list of past tense, infant learning, and generalization




Optional Readings


McClelland, J.L. & Rumelhart, D.E. (1991). An interactive activation model of context effects in letter perception: Part 1. Psychological Review, 5, 375-407.


McClelland, J.L. & Elman, J.L. (1986). Interactive processes in speech perception: The TRACE Model. In D.E. Rumelhart & J.L. McClelland (Eds.) Parallel Distributed Processing, Vol. II. Cambridge, MA: MIT Press.

Hare, M., & Elman, J.L. (1995). Learning and morphological change. Cognition, 56, 61-98.






Day 3: Language acquisition


        The poverty of the stimulus; lack of negative feedback

        Generalization vs. conservatism: Empirical findings

        What do infants and children know, and when do they know it?

speech perception; language identification; word segmentation; grammatical categories

        Infants and grammar learning

        (The past tense)


Required Readings

Bates, E., & Goodman, J. C. (1997). On the inseparability of grammar and the lexicon: Evidence from acquisition, aphasia, and real-time processing. Language and Cognitive Processes, 12, 507-584.



Marcus, G. F., Vijayan, S., Rao, S. B., & Vishton, P. M. (1999). Rule learning by seven-month-old infants. Science, 283(5398), 77-80.



Seidenberg, M. S., & Elman, J. L. (1999a). Do infants learn grammar with algebra or statistics. Science, 284, 434-435.



Seidenberg, M. S., & Elman, J. L. (1999b). Networks are not 'hidden rules'. Trends in Cognitive Sciences, 3(8), 288-289.



Optional Readings

Lewis, J., & Elman, J. (2001). A connectionist investigation of linguistic arguments from the poverty of the stimulus: Learning the unlearnable.



Gomez, R. L., & Gerken, L. (1999). Artificial grammar learning by 1-year-olds leads to specific and abstract knowledge. Cognition, 70(2), 109-135.



Plunkett, K., & Marchman, V. (1993). From Rote Learning to System Building - Acquiring Verb Morphology in Children and Connectionist Nets. Cognition, 48(1), 21-69.

        (http://crl.ucsd.edu/~elman/Bulgaria/plunkett-marchman-rote.pdf) [tech report version]


Plunkett, K., & Juola, P. (1999). A connectionist model of english past tense and plural morphology, Cognitive Science, 23, 463-490.






Day 4: Sentence processing: Experimental results and modeling


        Basic issues and phenomena in sentence processing

        The sausage machine and two-stage processor

        Constraint satisfaction / probabilistic / expectation generation models


Required Readings


Tanenhaus, M.K., & Trueswell, J.C., (1995). Sentence comprehension. In J.L. Miller and P.D. Eimas (Eds.) Handbook of Perception and Cognition, 2nd edition. Vol. 11: Speech, Language, and Communication. NY: Academic Press. Pp. 217-262.



McRae, K., Spivey-Knowlton, M.J., & Tanenhaus, M.K. (1998). Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38, 283-312.



McRae, K., Hare, M., Elman, J.L., & Ferretti, T. (2006). A basis for generating expectancies for verbs from nouns. Memory and Cognition, 33, 1174-1184.

        (http://crl.ucsd.edu/~elman/Bulgaria/ McRae_Hare_Elman_Ferretti-MemCog.pdf)




Optional Readings


Seidenberg, M.S., & MacDonald, M.C. (1999). A probabilistic constraints approach to language acquisition and processing. Cognitive Science, 23, 569-588.



Christiansen, M.H., & Chater, N. (1999). Connectionist natural language processing: the state of the art, Cognitive Science, 23, 417-437.




Day 5: Dynamical approaches & linguistic theory


        Sequential processing

        Simple (and other) recurrent networks

        The lexicon and grammar, rethought

        Dynamical analyses and usage-based grammars

        The role of event knowledge in language


Required Readings


Elman, J.L. (1990). Finding structure in time. Cognitive Science, 14, 179-211.



Elman, J.L. (draft). On the meaning of words and dinosaur bones: Lexical knowledge without a lexicon. (in draft format; please do not distribute or cite)\




Optional Readings


Elman, J.L. (1995). Language as a dynamical system. In R. Port and T. van Gelder (Eds.), Mind as Motion: Explorations in the Dynamics of Cognition. Cambridge, MA: MIT Press. Pp. 195-223.



Rodriguez, P., Wiles, J., & Elman, J.L. (1999). A recurrent neural network that learns to count. Connection Science, 11, 5-40.






Afternoon section meetings

Afternoon sessions will be used to explore specific topics in depth and to present additional material.




Students who take the course for credit will be asked to write a brief (5-7 page) paper that critical reviews one or more of the articles read in class, or to comment on other work that is related to the issues discussed in the class.