Shakespeare’s Vocabulary Considered Unexceptional

Shakespeare’s Vocabulary Considered Unexceptional

Summary

Shakespeare’s vocabulary is held to be extraordinary among writers. Its relative enormity is unquestioned in the popular and academic literature, bolstered by – and reflexively reaffirming – the peculiar status Shakespeare holds within our culture. 1

A few simple programs were written to analyse his and other writers’ works whose corpuses were of similar size to see how they compared.

What I discovered suggests that Shakespeare’s vocabulary, while far from small, is far from extraordinary among writers when size of corpus is taken into account.

Results

What I found is expressed in the two graphs below (click to view).

Each is a graph showing the relative size of corpus, number of unique word tokens, and number of unique word stems for various authors’ available works in Figure 1, or a sub-set of them for comparison in Figure 2.

Comparison of writers' vocabulary based on available works

Figure 1

Comparison of writers' vocabulary with edited corpuses

Figure 2

Each writer is represented by three numbers.

The first column is the size of the corpus examined (ie how many “real” word tokens there were in the texts I gathered together), and divided by 20 (so the numbers are comparable to the other data points). The “corpus” for each writer was the works I could download from the Project Gutenberg website. (Note that Joyce’s corpus here does not include Finnegans Wake). 2

The second column is the number of unique tokens found in the above corpus. 3

The third column is the size of the writer’s vocabulary based on the number of unique word tokens in the corpus. This is based on the number of unique stemmed word tokens. A stemmed word is a “root” form of a word that may have several distinct relations in other word tokens. A stemming algorithm reduces the words “fishing”, “fished”, “fish”, and “fisher” to the root word, “fish”. To determine the word stems I used a Porter stemming algorithm implemented in perl freely available on the web. 4

Analysis

The first thing to acknowledge is that Shakespeare’s vocabulary is larger than some other notable writers with similar-sized corpuses. It’s significantly larger, for example, than Dickens or Richardson.

At first glance it appears that Shakespeare’s vocabulary was markedly larger than Marlowe’s. However, taking a similar sized corpus of Shakespeare’s younger works, you can see that the vocabulary size for these works is almost identical to Marlowe’s. To test the hypothesis that Shakespeare’s vocabulary grew as he got older, a similar-sized corpus of his later works was examined. Again, the results were very similar.

Melville, with fewer words in Moby Dick than the younger Shakespeare, has a greater vocabulary than displayed there and in Marlowe’s works.

Milton is often cited as having a smaller vocabulary than Shakespeare, but this is also not borne out by the analysis. In fact, given the relatively small size of his available corpus, his vocabulary is very large indeed.5

Hardy – with a similar sized corpus – also shows a vocabulary not dissimilar to Shakespeare’s. Far more unique words than any other writer, even given his smaller corpus, and the only writer in the study with more than 20,000 stemmed words.

The vocabulary king among writers is Joyce, whose vocabulary towers over Shakespeare’s (Finnegans Wake was not included) even with a significantly smaller corpus.

Shakespeare’s vocabulary might be reduced further if we took out place and other names from his works, and removed the variant spellings more common in the era before standardized spelling.

Conclusion

The myth of Shakepeare’s unusually large vocabulary suggests that our view of Shakespeare has been warped by our veneration of his work. Rather than see him as an unusually successful writer whose works have remained popular over centuries, we have tried to make his literary abilities seem extraordinary too. Shakepeare is also said to have invented many words. Is this a myth too?

Does the size of a writer’s vocabulary matter? Isn’t it even more impressive that he managed to do so much with nothing more than the tools other writers possess?

It may also be worth researching further whether this analysis indicates that there is a difference in vocabulary size exhibited between playwrights (eg Shakespeare, Marlowe), poets (eg Milton, Shakespeare, Marlowe(?)) and novelists (eg Richardson and Dickens) or even whether the our intuitive understanding of the categories are aligned with writers’ displayed vocabularies.

Footnotes

1
“However, the single most remarkable feature about Shakespeare’s poetic language is his extraordinary vocabulary, his choice of particular words to convey particular emotional attitudes. Earlier I have had occasion to note that Shakespeare’s working vocabulary is enormous (about 25,000 words, more than twice as many as his nearest rival, John Milton)” Ian Johnston, “Studies in Shakespeare: Some Observations on Shakespeare’s Dramatic Verse in Richard III and Macbeth”, 1999, http://records.viu.ca/~johnstoi/eng366/lectures/poetry.htm

“Critics have long recognized that Shakespeare had an unusually large mental lexicon that was perhaps organized around particularly strong image-based mental models. […] Shakespeare’s almost uniquely rich use of language.” M. T. Crane, Shakespeare’s Brain: Reading with Cognitive Theory (Princeton NJ: Princeton University Press, 2000), 24

G. L. Brook, The Language of Shakespeare (London: Andre Deutsch, 1976), pp. 26-64

S. S. Hussey, The Literary Language of Shakespeare (New York: Longman, 1982), pp. 37-60

2
Works used in analysis:

Charles Dickens: “A Christmas Carol”, “Bleak House”, “Barnaby Rudge”, “David Copperfield”

Samuel Johnson: “Grammar of the English Tongue”, “Lives of the English Poets: Prior, Congreve, Blackmore, Pope”, “Notes to Shakespeare, Volume III: The Tragedies”, “Johnson’s Notes to Shakespeare Vol. I Comedies”, “Prefaces and Prologues to Famous Books”, “Preface to a Dictionary of the English Language”, “Preface to Shakespeare”
Thomas Hardy: “A Pair of Blue Eyes”, “The Mayor of Casterbridge”, “The Return of the Native”, “Tess of the D’urbervilles”, “Jude the Obscure”, “Far from the Madding Crowd”, “Return of the Native”

George Eliot: “Middlemarch”

Henry James: “The Bostonians”, “Portrait of a Lady”, “The Wings of a Dove”

James Joyce: “Dubliners”, “Ulysses”, “A Portrait of the Artist as a Young Man”

Christopher Marlowe: “Various minor poems”, “Dido, Queen of Carthage”, “Dr Faustus”, “Edward II”, “The Jew of Malta”, “Massacre at Paris”, “Tamburlaine the Great (part i, ii)”

Herman Melville: “Moby Dick”

John Milton: “Areopagitica”, “Milton’s Comus”, “Minor Poems by Milton”, “Paradise Lost”, “Paradise Regained”

Samuel Richardson: “Clarissa”

Shakespeare: “The Sonnets”, “A Lover’s Complaint”, “All’s Well That Ends Well”, “Antony and Cleopatra”, “As You Like It”, “The Comedy of Errors”, “Coriolanus”, “Cymbeline”, “Hamlet”, “Henry IV (parts i, ii)”, “Henry V”, “Henry VI (parts i, ii, iii)”, “Henry VIII”, “King John”, “Julius Caesar”, “King Lear”, “Love’s Labour’s Lost”, “Macbeth”, “The Merchant of Venice”, “Measure for Measure”, “The Merry Wives of Windsor”, “Midsummer Night’s Dream”, “Much Ado About Nothing”, “Othello”, “Richard II”, “Richard III”, “Romeo and Juliet”, “The Taming of the Shrew”, “The Tempest”, “Timon of Athens”, “Titus Andronicus”, “Toilus and Cressida”, “Twelfth Night”, “The Two Gentlemen of Verona”, “The Winter’s Tale”

Shakespeare Younger: “The Comedy of Errors”, “Henry VI (parts i, ii, iii)”, “King John”, “Richard III”, “Taming of the Shrew”, “Titus Andronicus”, “Twelfth Night”, “Love’s Labour’s Lost”, “Romeo and Julie”

Shakespeare Older: “The Sonnets”, “Cymbeline”, “Hamlet”, “Henry VIII”, “King Lear”, “Macbeth”, “Measure for Measure”, “Othello”, “The Tempest”, “Timon of Athens”, “The Winter’s Tale”, “A Lover’s Complaint”

Burton’s “Anatomy of Melancholy” was also analysed, but contains a great deal of Latin text interspersed, making his vocabulary anomalously large.

3
A “word” any set of contiguous non-space, non-punctuation characters or punctuation not beginning with a number. Possessives (“.*’s”) were removed. Words shortened with “.*’d” have been replaced with “ed”).

4
Porter, 1980, An algorithm for suffix stripping, Program, Vol. 14, no. 3, pp 130-137,
See also: http://en.wikipedia.org/wiki/Stemming

5
See footnote 1, Johnston.

This entry was posted in Technical, Uncategorized. Bookmark the permalink.

17 Responses to Shakespeare’s Vocabulary Considered Unexceptional

  1. Ricardo says:

    This is interesting, but wouldn’t it be better to create ratios of unique words and uniquely stemmed words by total number of words? Because each of these authors have written a different total number of words and without quantitatively taking that into account it is somewhat difficulty to know what is really going on.

    The other point is that you might want to compare them within time clusters, for example, by century. It would probably be unfair to compare Shakespeare writing in the 16th century, when access to books was still somewhat difficult, to Dickens in the 19th when libraries were abundant and therefore much easier to have a broad vocabulary.

    Notice that your final conclusion is not necessarily wrong, but your analysis is probably not enough to justify your conclusions. Cool idea though.

    • zwischenzugs says:

      All interesting points.

      I had thought of a counter-argument to your time thesis: that Dickens was a more prosaic writer writing simpler narrative/populist works (I believe Shakespeare was popular, but considered “clever” and wordy by the groundlings), which might explain why his vocabulary was less broad.

      I’m not convinced by the literacy arguments, as the oral culture in Elizabethan England was vibrant and dynamic, and drew on many sources, and Shakespeare himself was likely exposed to much literature and diverse regions in England. Evidence is hard to come by, though.

      Regarding the ratio point, I’m not sure whether such a number would be that helpful, since doubling the number of words in a corpus would likely not double the number of unique stemmed words. You can deduce it from the figures given though.

      Mostly my motivation was to show how easy it is to disprove the commonly-held notion that Shakespeare’s vocabulary was as special as his place in literary history.

  2. Petr says:

    You’re applying rules to measure the impact of a notorious and shameless rulebreaker.
    Shakespeare wasn’t a genius in the same way that, say, Einstein was… Where Einstein
    teased out the underlying rules and structure of reality. No. Shakespeare took everything
    apart and put it back together in new and fascinating ways.

    Shakespeare did invent many words, but he also invented new uses for existing words and
    there is, as I can see, no way in which your study accomodates this. He upset not only the
    vocabulary but also the grammar and indeed the structure of communication. He did
    this in the context of a visual medium (theater) in which we know (comparatively) little
    about stage directions and actors interpretation.

    So, a simple count of words doesn’t tell us much about Shakespeares impact.

    • zwischenzugs says:

      The article is not about Shakespeare’s impact. It’s about the supposedly extraordinary nature of his vocabulary. His impact is not diminished by his un-extraordinary vocabulary, so why have people insisted on it?

  3. Worthy L says:

    With the statistical difference being large, this sort of kills the Marlowe was Shakespeare theory.

    Very interesting article, thanks!

    • zwischenzugs says:

      Thanks!

      I was thinking it might (if anything) do the opposite – young Marlowe and Shakespeare have very similar vocabulary profiles, so if Marlowe had lived/written on, then he would have potentially displayed the same vocabulary.

      However, there’s plenty of other evidence to disprove the “Shakespeare was Marlowe” (or vice versa) thesis… vocabulary size itself probably wouldn’t tell us much.

  4. Pingback: Assyrian Dictionary Complete « Living Languages

  5. hopkinshughes says:

    With vocabulary as with some other things, size matters less than what a writer does with it, but what Shakespeare did with his is so impressive in comparison to the other writers mentioned, great as they are, that he can certainly lose a few points in the size department without it doing his overall achievement the slightest harm. I would think that the most important measurement would be the number of words for which he’s listed by the OED as first use. Yet even this requires exception, since he may simply have been the first to publish it, or his publication been the first to survive. I would venture to guess that most or all of the writers who came after him benefitted by his habit of creating new words from Latin and Greek roots. A better comparison than any of these later writers (yes, Marlowe came later) would be Francis Bacon, who partnered with Shakespeare (the poet, not the Stratford dude) in following the French Pléiade in their effort to create a vernacular literature for their respective native languages out of the roots of Latin and Greek. The sixteenth century saw all the nations of Europe engaged on this project, of which Bacon and Sir Philip Sidney are known to have been involved, and which we can see that Shakespeare was too by his very achievement. More on this at my blog:http://politicworm.com/oxford-shakespeare/why-shakespeare-matters/

    • zwischenzugs says:

      I believe the OED lists him as first use on so many words mainly because that’s where they were looking, and didn’t have the technology to cross-check with other texts. If I’m right, then as more works are digitized from that era his reputation in that area will diminish also.

      Not sure what you mean by “Shakespeare (the poet, not the Stratford dude)”? Was there a poet called Shakespeare of the time not from Stratford?

      Also unsure what you mean by “Marlowe came later”? They were contemporaneous, and Marlowe’s life ended before Shakespeare’s career was in full flower.

  6. Richard C says:

    It sounds as if proper nouns were included (i.e., capitalized words representing names of people, places, etc.). Wouldn’t the number of these vary with the time period, and with the author’s background? By Shakespeare’s vocabulary I’d like to know the number of building blocks for his literature as an indication of his flexibility in expressing himself. This would exclude proper nouns.

    • zwischenzugs says:

      I’m not sure the assumption that the number of proper names known by people changes over time is correct, any more than their vocabulary.

      • Richard C says:

        Wouldn’t a simple analysis of your data settle this point? By the way, I’m not trying to punch holes in your work: I found your study fascinating. Surely one sign of a good article is that it stimulates additional questions.

      • zwischenzugs says:

        I’m sure there are plenty of holes to punch :)

        I’m afraid time is the problem. If you follow the github link you’ll see that the raw data was flattened from the Gutenberg sources.

        So you’d have to get the sources again, determine which capitalized words are proper nouns and then perform the analysis. A non-trivial task…

  7. Pingback: Embiggen Your Vocabulary

  8. Finn Fordham says:

    It would be cool to add the vocabulary (or recognisable and recognised words) of FInnegans Wake to the Joyce tower. Hard work but interesting.

  9. Will Berard says:

    Further reading: http://rappers.mdaniels.com.s3-website-us-east-1.amazonaws.com/ – a lot of hippity hop rapping artists have a larger vocabulary than the Bard.

    I’m only an anglophile Frenchman, but I’ve got the impression Shakespeare is not so much praised for the size of his vocabulary than for the number of phrases he coined. Not all that “new words” stuff, but phrases made up of non-fancy words, stringed together, that have found their way into common parlance.

    I was surprised to find out recently that “neither here nor there” for instance, is from Othello…

    • zwischenzugs says:

      I’m also skeptical about those claims! How we are supposed to know that he coined these words and phrases I’ve no idea. I suspect it’s simply that the OED notes that it’s the first recorded use.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s