How not to test a hypothesis

In Part I of SC's review of George Lakoff's Moral Politics series, we laid out the theoretical precepts which underlie Lakoff's analysis of American politics. Here, we turn our attention to the research methods by which Lakoff develops the specific metaphors which he claims are relevant.

Introspection has long been a controversial technique for linguistic investigation. Chomsky has made heavy use of it ever since Syntactic Structures, and people working in numerous branches of syntax and semantics have adopted it as a valid technique for picking example sentences to test their theories. And there's a certain plausibility to it; when Chomsky wrote that his aim was "constructing a description, and, where possible, an explanation, for the enormous mass of unquestionable data concerning the linguistic intuition of the native speaker, often himself [Chomsky]" (Aspects of the Theory of Syntax, p. 20), it was hard to deny that native speakers ought to have a good idea of whether or not sentences in their language are grammatical.

That doesn't make their intuitions reliable to the level of accuracy we might hope for in constructing a genuinely scientific theory. In Chris Manning and Hinrich Schütze's wonderful textbook, Foundations of Statistical Natural Language Processing, the following sentences (among others) are presented right on page 9 as examples of sentences whose grammaticality has been judged by linguists in the process of making theoretical arguments. See if you can guess which one Henk van Riemsdijk (not a native English speaker) and Edwin Williams (who has no such excuse) jointly argued was grammatical:

1) John I believe Sally said Bill believed Sue saw.
2) What did Sally whisper that she had secretly read?

Yup, these highly-trained linguists, both possessed of Ph.D.s and sterling research reputations (SC has favorably cited Edwin Williams in a number of essays for classes), got their judgments exactly backwards compared to most of the rest of us. Of course, in the original context, it's possible that the sentences both came loaded with all sorts of indices and traces, and that it made more sense in theoretical context. But it's safest to be suspicious of a theory that insists that a sentence is ungrammatical only when larded with theory-internal diacritics that no native speaker will ever see.

Thus, many linguists -- especially those in the computational and historical branches (pride of place actually going to the latter) -- insist on doing their work with actual data in collections of text known as corpora. It is greatly annoying to SC that this position, which has a much longer and more respectable heritage than introspection, is actually the controversial and trendy one in modern linguistics. Of course, it's possible to go too far in this direction as well, as Geoff Pullum discovered when he and Rodney Huddleston published their monumental Cambridge Grammar of the English Language. But for the purposes of investigation, rather than simply drawing up summary conclusions, there's no doubt in SC's mind that good linguistics requires good data.

Linguistic introspection, however, is at least plausible on the grounds that native speakers of a language obviously have knowledge about the language, even if their attempts to formalize it are necessarily disastrous due to their limited perspective (the real reason that corpora are so important). It is considerably more difficult to maintain this plausibility when claiming to introspect about things that you have no such connection to. For example, SC's readers are aware ([at times, painfully so -- ed.]) that their host is Jewish. Without deeper knowledge of his educational background or religious training, they might do well to take his pronouncements about Jewish practices and beliefs cautiously, but they can be sure that, unless they're also Jewish, his statements at least are grounded in the sort of experience that George Lakoff calls critical (as we noted in part one) to developing moral reasoning or linguistic knowledge. Were SC to suddenly claim to have any kind of meaningful insight into how Catholics or Baptists understand the nature of a monotheistic G-d while simultaneously proclaiming the Trinity, and were your host to do so without providing evidence of serious research with footnotes by the dozen, readers would be notably more skeptical of his claims -- and rightly so. Introspection about things you aren't deeply acquainted with is not a serious research strategy.

But one can imagine doing even worse than that. Suppose your host claimed to make judgments about the internal moral reasoning of Nazis, while disclaiming any need to read Mein Kampf. Or judgments about Lenin while taking it as unnecessary to read What is to be Done?. One can easily judge their actions by one's own moral standards, and in the case of the great murderers of history, this is not terribly difficult to do. It is, however, considerably harder to justify claims to understanding their internal motivations without reading them, and equally hard to justify claims to understanding their symbolic language when one can't be bothered to actually watch their language in use.

Why, then, isn't Moral Politics simply written off as the ravings of an uninformed bigot?

In the course of two chapters detailing the contrasting models of morality which he claims underlie conservative and liberal thought, Lakoff introduces 18 explicit metaphors which he claims are features of both sides' thought processes. In those chapters, he groups them into three overarching categories, and then purports to demonstrate throughout the rest of the book that the differences in conservative and liberal reasoning fall out from a difference in their ranking. Much of the plausibility of this to linguists comes from its superficial resemblance to Optimality Theory, and other constraint-based approaches to studying language. The form of the argument is sound, because it is a form familiar to linguists, and has been applied successfully in numerous other environments.

The force behind OT, however, comes from the fact that its constraints are demonstrably grounded in -- here we go again -- actual data. *Coda and MAX don't simply arise as explanatory mechanisms because Alan Prince and Paul Smolensky sat around in an office thinking about their particular northeastern American English dialects. They were proposed to explain specific facts about real languages, and shown in demonstration after demonstration to explain even more languages. Lakoff's entire list of references of mainstream conservatism consists of two books on morality by William Bennett, a book on the Contract With America co-authored by Ed Gillespie, and two books of popular polemics by Newt Gingrich and Rush Limbaugh. Aside from Bennett's books, the rest aren't even cited in the body of Moral Politics. Lakoff provides a longer list of references for "neoconservatism", including 17 works, many of which are more serious tracts by scholars like Nathan Glazer and Seymour Lipset, but the actual evidence that they have been consulted in the body of his text is as skimpy as his other list. Lakoff's partitioning of the books into conservative/neoconservative is by itself reflective of ignorance about the actual status of "neoconservatives" within the broader conservative philosophy. Many of the authors that he cites as such would deny their membership in a "neo" group separate from mainstream conservatism. Lakoff's typology does, however, reflect a left-wing understanding of the varieties of conservative thought, a subject we'll take up in more detail in Part III. This isn't unreasonable -- Lakoff is scrupulous about making his own biases clear, and about trying to remain neutral in those parts where he is not explicitly speaking tutorially to liberals -- but it does reflect a larger failure to recognize and deal with his own internal conceptual organization.

Lakoff's efforts to research liberalism are only marginally better. He cites 11 works of "theoretical" (as distinct from "political" liberalism), including John Rawls and Richard Rorty. He also cites 14 works representative of the "communitarian" critique of Rawlsian liberalism most prominently associated with Amitai Etzioni, but the body of the work contains few actual references to these works. Lakoff points out that the references section of his book contains much that isn't cited, and that many of the listed items are meant only to be introductory/supplementary material. Assuming that Lakoff has, in fact, read all of the works in his list of references, his knowledge of theoretical liberalism is better than his knowledge of theoretical conservatism, but not by much.

Lakoff has two defenses against this line of attack. First is that his efforts are not critiques of what morality should be, but what it is, and that engaging the thinkers on either side would cloud the issue. To this end, he invokes the standard Good Housekeeping Seal of Linguistics, the prescriptive/descriptive injunction. SC will let Lakoff speak for himself here:

The models we are discussing are descriptive, not prescriptive. They are attempts to describe what people's actual unconscious worldviews are, not what they should be. Most theories of liberalism and conservatism are not concerned with description but with prescription. For example, John Rawls's [sic] celebrated theory of liberalism is not an empirical descriptive study, but an attempt to characterize a prescriptive theory of justice, from which liberalism follows. As a descriptive account of actual liberal political stands on issues, it is a failure, as we shall see. My job here is to describe how people make sense of their politics, not how they should. (p. 37)

This is a fine defense of how things ought to be in linguistic research, but Lakoff is so careful to avoid making judgments about the prescriptions that he simply relies on introspection to see how people make judgments about what they think. There is not a single citation of any of the popularizing media for conservative or liberal ideas, i.e. magazines. An analysis of how either side expresses their underlying morality might benefit greatly from even the slightest acquaintance with National Review, The New Republic, or even In These Times. When discussing the question "How can you love your country and hate your government?", Lakoff would do far better to cite Ronald Reagan's speech where he stated "Government is not the solution to our problem. Government is the problem." than what he actually does, which is to provide two pages of introspection on what a Ku Klux Klansman must think. Examples of actual rhetoric are almost wholly absent from a book purporting to analyze the deeper motivations of political rhetoric.

Lakoff has an answer to that objection, too, the second of his defenses for an introspection-based book on how other people think. It's just too hard to do this sort of study the "right" way. Lakoff argues:

Psycholinguistic testing has begun to be able to discern the existence of conceptual metaphors in cognitive models, but no experimental paradigms of the complexity needed to test this hypothesis now exist (see References, A1, Gibbs 1994). Survey research has not yet developed an adequate methodology to test for the presence of complex metaphorical cognitive models such as these. (p. 158)

This is a diversion, and Lakoff knows it. It may well be true that survey research hasn't developed a technique for determining whether models organized according to the theories laid out in his earlier works exist, and since SC hasn't conducted a serious piece of survey research since his 1998 bachelor's thesis, he'll refrain from comment on that point. But the individual metaphors are very easy to test. It is not difficult to imagine laying out a survey full of actual statements by liberals and conservatives making use of the metaphors that Lakoff proposes, and getting people to rate them on a 5 or 7 point scale as being more conservative or more liberal. It's also not difficult to imagine establishing experimental controls for the demographics and biases of the survey-takers; there are plenty of scales accepted by psychologists for diagnosing conservatism. This would go a long way towards establishing the psychological reality of the metaphors -- even without additional evidence for the specific details of their organization, it would provide confirmation that they're the theoretical primitives that should be getting attention. This is especially important for Lakoff's case since he claims that we're all working from the same metaphors, we just prioritize them differently.

Instead, Lakoff asks us to accept that this is an exercise in "cognitive modeling" (p. 156). We should accept his word that his models account for: 1) "why conservative and liberal political stands group together" (even though he hasn't demonstrated that he knows what those stands are), 2) "what puzzles liberals about conservatives and conservatives about liberals" (even though he doesn't demonstrate actual examples of said puzzlement), and 3) "the details of conservative and political discourse" (which neither the body of his text nor his references suggest he has made anything like a serious study of). (All quotes taken from p. 157.) Geoff Pullum's caveat about corpus data does not apply here -- it would be fine for Lakoff to give generic examples of political language in his summation, if he had already demonstrated that political language actually works like he claims.

As we'll see in Part III (after the weekend; SC may update, but not on this topic), Lakoff's understanding of conservatism comports very poorly with actual conservative rhetoric, to say nothing of his "grouping of political stands". We'll do the same for liberalism in Part IV, where Lakoff will come off a bit better due to his being a "native speaker" -- but not by as much as he'd think.

Comments

[Rachel]

If you add commas to sentence 1 ("John, I believe, Sally said Bill believed Sue saw") it sounds better. It's still a confusing sentence that no one would ever say, but I don't think it's exactly ungrammatical.

[Rick]

You said two things which seem contradictory to me:
[quote]

Introspection about things you aren't deeply acquainted with is not a serious research strategy.
...
This is a diversion, and Lakoff knows it.
[/quote]

So you have critical experience about what Lakoff knows to be a diversion, and you got it through introspection -- if I understand you correctly. It looks to me like Lakoff could turn your criticisms around and use them against you.

Then again, it could be that you have a substantial point that I'm missing. The technical side of this may be obvious to professional linguists, and maybe I lack the basic skills to catch your drift.

This is a good example of why I won't believe in any logic more subtle than the formula satisfiability problem. Once it gets past first-order logic it all goes blurry to me.

P.S.: Sentence 1 sounds perfectly natural to me when I say it out loud. If it had commas it would be as grammatical as anything in Michael Swan's _Practical_English_Usage_.

[Semantic Compositions]

I'll deal with the easy one first -- sentence 1 sounds completely awful to me, but the thing that makes introspection about grammaticality so much fun is that people with only slightly different dialects can disagree greatly. The point there was that more native English speakers will disagree with van Riemsdijk and Williams than not about the judgment, and the failure of such judgments to be reliably universal is what makes them inappropriate as a research technique.

Moving on to your more substantive point, there's no contradiction in what I said otherwise. One of these statements was a rhetorical point, and the other was not. Lakoff has argued vehemently against introspection as a research method, and some of his other writings provide excellent case studies in how to do empirical research. The comment about "and Lakoff knows it" was intended to be emphatic, but it's well grounded in things he has actually said -- including things cited in the body of this post. Quoting Lakoff hardly counts as introspecting about his mental states. Drawing conclusions from those quotes doesn't make me a hypocrite.

But your comment indicates that you think I'm radically skeptical of our ability to discern other people's mental states, to the extent that they're almost completely unknowable. I don't think that's a reasonable interpretation of my comments; the standard I set in judging Lakoff's work isn't that one must totally immerse oneself in the life's work of one's subjects, but that if you're not a "native thinker", then you had better "[provide] evidence of serious research with footnotes by the dozen". Lakoff simply doesn't show that he's bothered to read even a slightly representative sampling of conservative rhetoric, and his treatment of liberal speech is not notably better. I actually read Lakoff's books before making any judgments about his thinking, and quotes with footnotes can be found in every section of the review. That's a compliment Lakoff doesn't pay to those whose speech he purports to explain.

[Rick]

But your comment indicates that you think I'm radically skeptical of our ability to discern other people's mental states, to the extent that they're almost completely unknowable. I don't think that's a reasonable interpretation of my comments;

No, actually, that's not what I said, nor what I meant. I don't know whether the only problem is that I suck at articulating or whether there are other problems.

This is why I stick to the formula satisfiability problem.

[MickM]

The academic in the back of my mind is fascinated by your critique of Lakoff's generalizations about the essential frames of political ideology. I came to Lakoff through Benjamin Lee Whorf and Keith Basso, not politics, and found his theories and assertions fascinating but, shall we say, incomplete.

My dissatisfaction with Lakoff's work generally is for the lack of culturally diverse substantiation in his assertions about metaphor. The topic is crying for some basic data comparable to what you find with a quick foray into, say, ethnomathematics.

The tactician in the front of my mind, however, is not persuaded that Lakoff is wrong is his generalizations, for a wonderfully simple reason. I suddenly find myself able to make sense of conservative rhetoric and even, thank Dog, to dissect, rebute, and confute it. That clarifying effect may be the result of felicitious overgeneralization, but it's getting the job done. You can tighten screws with a butter knife, if that's all you've got.

Great discussion. I'll be back.
Mick