By: Peter Hagoort, William Matchin, Greg Hickok, Natalia Levshina, & Dirk den Ouden
Introduction
On March 25 of this year (2020) I presented my on-line lecture “Beyond the language given: Language processing from an embrained perspective” in the wonderful C-STAR lecture series, organized and moderated by Dirk den Ouden. In the aftermath of my lecture a series of email exchanges happened between a small number of colleagues who had listened to my talk, and myself. The purpose was to clarify seemingly divergent positions about the neurobiological underpinnings of mainly syntactic operations. In my view this further discussion helped to clarify the different opinions. Julius Fridriksson took the initiative to make this exchange of ideas available to the wider community of brain and language researchers. As you will see, there is both convergence and divergence of opinions. Convergence exists about the view that both left frontal and posterior temporal cortex areas play a role in syntactic processing, about the lexicalist nature of aspects of syntactic processing, about different (degrees) of recruitment of these syntax-relevant areas in language production and language comprehension. Divergence exists about the distribution of labour and the exact contributions of temporal and frontal cortex areas to syntactic processing in production and comprehension. An important aspect in this debate is the available evidence. We are all faced with some inconsistencies between what data from aphasic patients seem to show and what we find in neuroimaging experiments with healthy participants. It is a challenge for the field as a whole to see how these seemingly inconsistent pieces of information can be reconciled. Nevertheless, although the details matter it is a reassuring fact that overall converging forces seem to have gotten stronger over the recent years. Of course, it is up to you to see if you can agree with my optimistic note.
Peter Hagoort
Nijmegen, August 3, 2020
From: Hagoort, P. (Peter)
Sent: Monday, June 8, 2020 4:41 PM
To: William Matchin
Dear William,
After my C-Star lecture you was so kind to send me your Cerebral Cortex paper
and the Agrammatism and Paragrammatism paper. I promised to send you my
thoughts once I have read these papers. This took a while longer than I had
planned. But managing the consequences of the COVD-19 pandemic for my research
institutes took most of my time and energy in recent weeks.
That being said, I have now read your papers with great interest, and would like to share some
of my thoughts. There might be more convergence than you think between our views, but also
some divergences. Let me point out the convergences first.
(i) We both agree that pMTG and LIFG are crucially involved in syntactic processing. Although you suggest in your Agrammatism/Paragrammatism paper that I single out the LIFG as the area for syntactic function (p.4), this is a position I never defended. I have always stressed the importance of both pMTG and LIFG and their connections.
(ii) You defend a lexicalist position referring to Jackendoff, Joshi and Schabes, Vosse and Kempen, etc. I agree completely. In fact, to the best of my knowledge I was the first to introduce that framework into the neurobiology of language and link it to the distribution of labour between pMTG and LIFG. See attached my paper in NeuroImage (2003) and Neuroscience and Biobehavioral Review Paper (2017) where this is integrated into an overarching perspective on the whole system. Since these lexical-syntactic templates are hierarchical, I agree that pMTG contributes to the hierarchical component of syntactic computation. What I did not understand is the quote on p.6 in your Cerebral Cortex paper: “Unlike some theorists (Hagoort, …) we do not propose an anatomical correlate of basic syntactic operations” I must have missed something here, since I thought you proposed a cortical model of syntax, with some very similar basic ingredients.
(iii) In fact I share the view that Broca’s area activation is driven by working memory requirements for language and other cognitive domains. This is what is needed to Unify the hierarchical syntactic templates retrieved from the lexicon from pMTG areas. For this we have observed evidence in among others the paper by Snijders et al (2007).
(iv) As I have argued in e.g. my 2019 Science paper, LIFG is very likely not language-specific. Unification operations happen in multiple domains of cognition. We find, for instance, overlapping activations in language and music in Broca’s area. This is very much in line with Ray Jackendoff’s view in his recent book on morphology: “Unification is a plausible combinatorial principle not only for linguistic structure, but also for a variety of other cognitive domains” (p.29). I agree. Hence LIFG is not a syntax area. It contributes to syntax only in its connection to pMTG, to instantiate unification operations between hierachical pieces of syntax retrieved from pMTG (which is domain-specific). LIFC on its own is not a language area. It becomes a language area in dynamic interaction with domain-specific areas in temporo-parietal cortex.
(v) I share the view that the LIFG is more strongly activated in production than comprehension. We also see that. But probably for another reason than linearisation. I think it is because we can bypass the computation of a full phrasal configuration in comprehension (good-enough processing), but not in production. Anyone, like yourselves, with experience in communication with aphasic patients has to agree that phonology, semantics and word meanings are way more important than syntax. Usually there are sufficient context cues to get the right interpretation without syntax. As a consequence of Chomskyan syntactocentrism, syntax is highly overrated in comparison to semantics and phonology. In comprehension we can largely do without syntax. But not so in production. Hence the stronger recruitment of LIFG in production to enable the production of a grammatically well-formed utterance, in which the correct unifications have to be made.
(vi) You seem to make a distinction between hierarchical syntax and morphosyntax. With Jackendoff’s recent view, I agree that it is hard to make such a distinction. I believe that, although at a different time-scale, the same basic operation of (de)composition happens in morphology between e.g. a stem and an inflectional morpheme. Unification all the way down.
(vii) Remains the issue of agrammatism vs paragrammatism. Here on the one hand you refer to earlier works showing that lesions in Broca’s area do not result in lasting Broca’s aphasia, but elsewhere you connect Broca’s area to a role in production. This sounded to me somewhat inconsistent, but maybe I missed the point here. I accept that these symptoms are quite different, and largely related to a difference in lesion localisation. But without explicit modeling it is hard to see what drives the differences. Let me make two suggestions. The first one is that in my experience of testing a few hundred aphasic patients, the patients with a posterior lesion far more often experience some form of anosognosia. They are less able to monitor their own output, and adapt their speech accordingly. This is why you see more adaptation symptoms in agrammatics as has been argued by Kolk and Heeschen. The other possible reason is the following: The syntactic templates (treelets) are retrieved based on the lexical-conceptual input during sentence planning. This information is clearly harder to access in the posterior patients. Hence the ctivation of the correct treelets might also be affected resulting in misselections and underactivations. This might result in the paragrammatic “sentence monsters”. Without explicit modeling it is hard to prove thta this might be an explanation. But at the same time, the double dissociation of the symptoms does not prove a different architecture for comprehension and production.
Well I guess for now this will do. Most happy to hear what your think.
Keep yourselves safe and healthy
best wishes,
Peter
________________________________________
From: MATCHIN, WILLIAM
Sent: Monday, June 8, 2020 11:40 PM
To: Hagoort, P. (Peter)
Cc: Greg Hickok; DEN OUDEN, DIRK; FRIDRIKSSON, JULIUS
Subject: Re: Reply
Hi Peter,
Thank you very much for this feedback – it was kind of you to respond in such detail. I agree, there is more convergence between our theories than among others, particularly with respect to the lexicalist framework. We should have highlighted this convergence more in the paper than we did.
I think one generally important issue is that the term “syntax” and “syntactic processing” are often too vague to be useful. Is it fair to say that you believe the primitive syntactic operation “unify” corresponds to (subregions of) Broca’s area? If so, then this is perhaps the primary difference between our framework and yours, in that we do not think any basic compositional syntactic operation resides in Broca’s area, or indeed does not really correspond to any cortical area at all.
On this point, I think we both might have similar explanations of paragrammatism – disruption of stored syntactic “treelets” as well as conceptual-semantic representations. But where we would strongly differ, I suspect, concerns agrammatism. There is a bit of tension about this, because while lesions *restricted* to Broca’s area do not cause Broca’s aphasia (or agrammatism), agrammatism is associated with damage to Broca’s area. These are simply the data, which present a problem for anyone trying to figure out the function of Broca’s area and how syntax is implemented in the brain. But we strongly think there is no primitive syntactic operation associated with Broca’s area, and that agrammatism does not reflect the loss of the ability to combine lexical items into coherent syntactic structures with semantic interpretation. So I think this is the core difference in our views. Do you think that’s accurate?
Thanks again,
William
________________________________________
From: Greg Hickok
Sent: Thursday, June 9, 2020 5:39 PM
To: Hagoort, P. (Peter)
Cc: MATCHIN, WILLIAM; DEN OUDEN, DIRK; FRIDRIKSSON, JULIUS
Subject: Re: Reply
Hi Peter,
I’ll give my thoughts. William’s may differ.
Thanks again for the clarification. It helps a lot to understand this. My view is not that something like Merge or Unify don’t exist, this has to happen as you note, but that it happens in various ways and in various networks. Treelet’s have to be unified in comprehension (PTL) and sequences of words and syllable have to unified in production (IFG+PTL). I reject the idea that says Broca’s is THE hub for an operation like Unify.
If you only look at functional data in healthy people one can make a reasonable argument for compression-production “intertwinement”, but the neuropsych data tell a different story. Damage to Broca’s area and surrounds does not cause the kind of problems with sentence comprehension that fronto-centric models predict.
Frontal-based prediction models of comprehension in any strong form are computationally ill-conceived (see Ch. 10 of my book) and empirically unsupported. We estimated the frontal contribution to speech perception to be 1-2 dB under noisy listening conditions for some items. Ie, a negligible effect:
https://link.springer.com/article/10.3758/s13423-019-01580-2
-g
________________________________________
From: Hagoort, P. (Peter)
Sent: Wednesday, June 10, 2020 2:23 PM
To: William Matchin
Hi William,
A quick reply to you. I will answer Greg’s comments later.
I don’t think of Unification as a core (or primitive) syntactic operation.
It is a domain-general capacity, related to the Working Memory capacities
of LIFC, to have access to domain-specific information in multiple areas
of temporo-parietal cortex with the possibility (in dynamic interaction
with the domain-specific areas) to build larger (phonological, semantic, syntactic,
musical, mathematical, full words out of morphological stems and affixes) stuctures out of the basic building blocks.
I wouldn’t call that a primitive linguistic operation (e.g. in contrast
to Merge; I explicitly don’t call it Merge, and my position is quite
different from Angela Friederici’s view).
In general, I find it hard to see how one can do without such a
function, since we all agree that we do not retrieve full-blown sentences
out of long-term memory (or the mental lexicon, however you want
to call it), but compute them on the fly. This is the famous “infinite use
of finite means” aspect. Hence, the lexicon cannot be the whole story.
In that sense the concept Unification is quite theory-neutral; it is not
committing itself to any specific linguistic theory.
In that context, it is not yet fully clear where you see this contribution arising,
and what exactly the contribution of Broca’s area is (e.g. Gary Dell’s model
states that in comprehension prediction is production; in many ways the systems are
more intertwined than was assumed in classical models of psycholinguistics)
I am happy to hear how you see the convergence/divergence in relation to
my, hopefully, further clarification.
best wishes,
Peter
________________________________________
From: Hagoort, P. (Peter)
Sent: Wednesday, June 17, 2020 12:52 PM
To: Greg Hickok
Hi Greg,
Thanks again for clarification.
A further clarification from my side.
I am in agreement with your paper, and I don’t think the motor areas play a necessary role
in speech perception. The motor theory of speech perception is untenable.
Your book on mirror neurons was convincing and, for the record, attached a paper
in which we came to the same conclusion.
Hence, full agreement at the level of analysis of the input (speech) and the output
(phonetic encoding, articulation).
But the Unification I am referring to is related to core operations such as syntactic
encoding and parsing.
The TMS studies of Watkins and others in support of the claim for the involvement
of motor systems in speech perception usually target ‘real’ motor or premotor areas,
not necessarily Broca’s area. Hence, we can debate to what extent Broca’s area can be called
a motor area, although monkey F5 is claimed to be its homologue.
best wishes,
Peter
________________________________________
From: Greg Hickok
Sent: Wednesday, June 17, 2020 11:46 PM
To: Hagoort, P. (Peter)
Cc: MATCHIN, WILLIAM; DEN OUDEN, DIRK; FRIDRIKSSON, JULIUS
Subject: Re: Reply
But the Unification I am referring to is related to core operations such as syntactic
encoding and parsing.
Thanks Peter. I think this is the one point to disagreement. So, let’s talk about what the different views predict. During comprehension, I would predict that you can do without frontal regions even when full blown hierarchical tree structures need to be constructed as long as working memory demands are minimized. I think you would predict that without frontal Unification-related regions, the system could not achieve full syntactic representations and therefore would have to rely on good-enough parsing (lexical semantics and some crude heuristics). Is that a fair characterization?
Assuming so, the trick is to find stimuli that you would agree require full syntactic representations for comprehension and I would agree minimize working memory enough. William and I consider passives a good example of such a case. Would you agree?
________________________________________
From: Hagoort, P. (Peter)
Sent: Tuesday, June 30, 2020 11:35 AM
To: Greg Hickok
Hi Greg,
It would indeed be nice if a clear case could be found. However, I am afraid passives won’t do.
With Culicover & Jackendoff (Simpler Syntax, 2005) I hold the view that a passive is “a piece of structure in its own right that can be unified with the other independent pieces of the sentence” (Simpler Syntax, p. 203).
A check of the Alpino corpus for Dutch by Natalia Levshina in my group has shown that in Dutch worden + Past Participle + door always conveys a passive meaning.
Worden + Past Participle is always passive (one occurrence of the impersonal passive). the preposition door + NP in combination with worden + Past Participle always
introduces the agent or the cause of a passivized event.
Hence, I don’t think that an alternation like active/passive would be a clear test case for finding out if there’s syntactic processing in LIFG. The reason is that there are surface cues
for passives like worden and door (for the case of Dutch; similar cases can be found in other languages). Upon encountering these words in the input string a lexically specified template
for passive is immediately retrieved from temporal cortex areas that in my view support the lexical component of syntactic processing.
Hence a passive can be recognized without the involvement of the LIFG.
best wishes,
Peter
________________________________________
From: Greg Hickok
Sent: Tuesday, June 30, 2020 10:50:49 PM
To: Hagoort, P. (Peter)
Cc: Greg Hickok; MATCHIN, WILLIAM; DEN OUDEN, DIRK; FRIDRIKSSON, JULIUS; Levshina, Natalia
Subject: Re: Reply
I can see some sense in that argument. So, would you agree that Comprehension of object extracted relative clauses should require IFG?
-g
________________________________________
From: Levshina, Natalia
Sent: Thursday, July 2, 2020 8:43 AM
To: Greg Hickok
Dear Greg (if I may),
Dear all,
if I may share my opinion in this fascinating discussion, I think object-extracted relative clauses (RCs) are still somewhat problematic. There is no clear evidence that the former require more effort than subject-extracted RCs if one takes into account language users’ discourse-based expectations. According to Fox and Thompson (1990), object-extracted RCs usually have inanimate heads and discourse-given pronominal subjects (e.g. “the book you’re reading”). This is a consequence of so-called preferred argument structure in discourse (Du Bois 1987). Experiments where these conditions are satisfied (e.g. Mak et al. 2002, 2006; Kidd et al. 2007) do not show that object-extracted RCs are more challenging than subject-extracted RCs for adults or children. Verb semantics matters, too.
In general, it seems to me that’s is very difficult to find a clear case of syntactic processing that doesn’t involve any lexical, semantic or pragmatic cues that could help language users to recognize the structure. I’d be very curious to hear any ideas about how to overcome this obstacle!
All the best,
Natalia
________________________________________
From: MATCHIN, WILLIAM
Sent: Thursday, July 2, 2020 8:50 AM
To: Levshina, Natalia
Cc: Greg Hickok; Hagoort, P. (Peter); DEN OUDEN, DIRK; FRIDRIKSSON, JULIUS
Subject: Re: Reply
Hi Natalia,
Thanks for jumping in here. The question is not whether object-relative clauses are harder than subject-relative clauses, in general. The question is: what is a task for a person with aphasia that would require them to use syntax (i.e., word order cues to structure) to perform adequately? The virtue of object- and subject-relatives is that an experiment can be made such that they involve the exact same lexical items; thus any differences in sentence interpretation between the two must come from analyzing the word order in some fashion to build a different structure. People with aphasia are often clever at solving tasks, so for instance interpreting an active sentence can be done by guessing the meaning based on (1) the lexical items and (2) a strategy in which the first NP is understood to be the agent. But object-relatives (vs. subject-relatives) eliminate this possibility, as the extracted object (which appears as the main clause subject, first in the sentence) cannot be solved using this strategy.
This is the logic behind Caramazza & Zurif (1976). Greg and I both agree with you quite heartily on the use of this contrast in *healthy* subjects, i.e. we don’t necessarily think that increased brain activity in healthy subjects for object-relatives vs. subject-relatives reflects syntactic resources per se.
-William
________________________________________
From: DEN OUDEN, DIRK <OUDEN@mailbox.sc.edu>
Sent: Thursday, July 2, 2020 3:13 PM
To: MATCHIN, WILLIAM; Levshina, Natalia
Cc: Greg Hickok; Hagoort, P. (Peter); FRIDRIKSSON, JULIUS
Subject: Re: Reply
Hi all,
Just to add two cents to this excellent discussion (and thanks for keeping me cc-ed!) – No one should be married to the idea that the neural differences elicited by processing OR vs SR sentences necessarily reflect use of syntactic resources, but it does depend somewhat on one’s definition of ‘syntactic resources’ – which is of course partly what this discussion is about. In contrast to Natalia, I do not believe there is any doubt that OR sentences are ‘harder’ than SR sentences, all else being equal, both for unimpaired speakers and for speakers with aphasia (across the board, by the way, so it’s not necessarily a marker of ‘agrammatism’). This evidence comes precisely from well-controlled experiments with reversible sentences in which the animacy of the arguments is not predictive of their thematic roles, and in which the same verbs are used in both conditions – basically everyone (and certainly at the group level) struggles more with the OR sentences (longer RTs, more errors) and this coincides with increased local brain activation associated with processing ORs. I think one can debate what that means, but not really whether these observations are accurate, even if it is possible to counter the general difficulty with ORs through syntactic priming or otherwise.
Cheers, Dirk
________________________________________
From: Hagoort, P. (Peter)
Sent: Monday, July 6, 2020 3:35 PM
To: DEN OUDEN, DIRK; MATCHIN, WILLIAM; Levshina, Natalia
Cc: Greg Hickok; FRIDRIKSSON, JULIUS
Subject: RE: Reply
Hi Dirk,
I am less convinced than you seem to be that SRs are always easier to process than ORs.
There is evidence that in other languages (e.g., Chinese) this is not the case.
Another example is Basque such as in the study by Carreiras et al (see abstract):
“Studies from many languages consistently report that subject relative clauses (SR) are easier to process than object relatives (OR). However, Hsiao and Gibson (2003) report an OR preference for Chinese, a finding that has been contested. Here we report faster OR versus SR processing in Basque, an ergative, head-final language with pre-nominal relative clauses. A self-paced reading task was used in Experiments 1 and 2, while ERPs were recorded in Experiment 3. We used relative clauses that were ambiguous between an object or subject-gap interpretation and disambiguated later in the sentence. The results of Experiments 1 and 2 showed that SR took longer to read than OR in the critical disambiguating region. In addition, Experiment 3 showed that SR produced larger amplitudes than OR in the P600 window immediately after reading the critical disambiguating word. Our results suggest that SR are not universally easier to process. They cast doubts on universal hypotheses and suggest that processing complexity may depend on language-specific aspects of grammar.”
groeten,
Peter
________________________________________
From: DEN OUDEN, DIRK
Date: Monday, July 6, 2020 at 4:10 PM
To: Hagoort, P. (Peter); MATCHIN, WILLIAM; Levshina, Natalia;
Cc: Greg Hickok; FRIDRIKSSON, JULIUS
Subject: Re: Reply
Hi Peter,
Thanks – and my apologies: my reply was unwittingly anglocentric in that I wasn’t even thinking about languages with fundamentally different syntactic structures – I was just assuming we were only talking about English and its canonical word order. Carreiras et al (2010) rightly point out that there are two types of theories – one that says the Subject-Relative preference is universal, regardless of the language in question (and to echo Natalia’s point, I assume regardless of discourse preferences), and the other says that the widely observed SR-preference DOES depend on the syntactic structure of the language itself. Carreiras et al, based on their experiment, argue that the SR vs OR preference can indeed be language-specific, and that seems only logical to me. I agree with that, as I agree with the fact that the SR preference in English can be countered through discourse manipulation. There is no reason to believe that subject-relatives are universally harder, regardless of their syntactic structure. In English, though, all else being equal, Object-Relatives are harder than Subject-Relatives, and it is not outrageous to make reference to the difference in syntactic structure between these sentences, to account for that widely observed discrepancy. So, that might mean that relatives ARE potentially a way in which to test ‘syntactic processing’, in that Object-Relatives can be assumed to require something ‘extra’ for their processing, which might well be syntactic in nature, no?
best, Dirk