Introducing spaCy

spaCy is a new library for text processing in Python and Cython. I wrote it because I think small companies are terrible at natural language processing (NLP). Or rather: small companies are using terrible NLP technology.

Update (October 3, 2016)

This post shows the original launch announcement for spaCy, which came with some usage examples and benchmarks. The benchmarks are quite out of date, but I’m pleased to say usage has changed relatively little. I spent a long time on spaCy’s initial design before this announcement. The pay-off has been that the API has remained pretty stable.

To do great NLP, you have to know a little about linguistics, a lot about machine learning, and almost everything about the latest research. The people who fit this description seldom join small companies. Most are broke — they’ve just finished grad school. If they don’t want to stay in academia, they join Google, IBM, etc.

The net result is that outside of the tech giants, commercial NLP has changed little in the last ten years. In academia, it’s changed entirely. Amazing improvements in quality. Orders of magnitude faster. But the academic code is always GPL, undocumented, unuseable, or all three. You could implement the ideas yourself, but the papers are hard to read, and training data is exorbitantly expensive. So what are you left with? A common answer is NLTK, which was written primarily as an educational resource. Nothing past the tokenizer is suitable for production use.

I used to think that the NLP community just needed to do more to communicate its findings to software engineers. So I wrote two blog posts, explaining how to write a part-of-speech tagger and parser. Both were well received, and there’s been a bit of interest in my research software — even though it’s entirely undocumented, and mostly unuseable to anyone but me.

So six months ago I quit my post-doc, and I’ve been working day and night on spaCy since. I’m now pleased to announce an alpha release.

If you’re a small company doing NLP, I think spaCy will seem like a minor miracle. It’s by far the fastest NLP software ever released. The full processing pipeline completes in 20ms per document, including accurate tagging and parsing. All strings are mapped to integer IDs, tokens are linked to embedded word representations, and a range of useful features are pre-calculated and cached.

Computers don’t understand text. This is unfortunate, because that’s what the web is mostly made of.

If none of that made any sense to you, here’s the gist of it. Computers don’t understand text. This is unfortunate, because that’s what the web almost entirely consists of. We want to recommend people text based on other text they liked. We want to shorten text to display it on a mobile screen. We want to aggregate it, link it, filter it, categorise it, generate it and correct it.

spaCy provides a library of utility functions that help programmers build such products. It’s commercial open source software: you can either use it under the AGPL, or you can buy a commercial license under generous terms.

Example functionality

Let’s say you’re developing a proofreading tool, or possibly an IDE for writers. You’re convinced by Stephen King’s advice that adverbs are not your friend so you want to highlight all adverbs. We’ll use one of the examples he finds particularly egregious:

>>> import spacy.en
>>> from spacy.parts_of_speech import ADV
>>> # Load the pipeline, and call it with some text.
>>> nlp = spacy.en.English()
>>> tokens = nlp(u"‘Give it back,’ he pleaded abjectly, ‘it’s mine.’", tag=True, parse=False)
>>> print u''.join(tok.string.upper() if tok.pos == ADV else tok.string for tok in tokens)
u‘Give it BACK,’ he pleaded ABJECTLY, ‘it’s mine.’

Easy enough — but the problem is that we’ve also highlighted “back”. While “back” is undoubtedly an adverb, we probably don’t want to highlight it. If what we’re trying to do is flag dubious stylistic choices, we’ll need to refine our logic. It turns out only a certain type of adverb is of interest to us.

There are lots of ways we might do this, depending on just what words we want to flag. The simplest way to exclude adverbs like “back” and “not” is by word frequency: these words are much more common than the prototypical manner adverbs that the style guides are worried about.

The Lexeme.prob and Token.prob attribute gives a log probability estimate of the word:

>>> nlp.vocab[u'back'].prob
-7.403977394104004
>>> nlp.vocab[u'not'].prob
-5.407193660736084
>>> nlp.vocab[u'quietly'].prob
-11.07155704498291

(The probability estimate is based on counts from a 3 billion word corpus, smoothed using the Simple Good-Turing_ method.)

So we can easily exclude the N most frequent words in English from our adverb marker. Let’s try N=1000 for now:

>>> import spacy.en
>>> from spacy.parts_of_speech import ADV
>>> nlp = spacy.en.English()
>>> # Find log probability of Nth most frequent word
>>> probs = [lex.prob for lex in nlp.vocab]
>>> probs.sort()
>>> is_adverb = lambda tok: tok.pos == ADV and tok.prob < probs[-1000]
>>> tokens = nlp(u"‘Give it back,’ he pleaded abjectly, ‘it’s mine.’")
>>> print u''.join(tok.string.upper() if is_adverb(tok) else tok.string for tok in tokens)
‘Give it back,’ he pleaded ABJECTLY, ‘it’s mine.’

There are lots of other ways we could refine the logic, depending on just what words we want to flag. Let’s say we wanted to only flag adverbs that modified words similar to “pleaded”. This is easy to do, as spaCy loads a vector-space representation for every word (by default, the vectors produced by Levy and Goldberg (2014)_). Naturally, the vector is provided as a numpy array:

>>> pleaded = tokens[7]
>>> pleaded.vector.shape
(300,)
>>> pleaded.vector[:5]
array([ 0.04229792,  0.07459262,  0.00820188, -0.02181299,  0.07519238], dtype=float32)

We want to sort the words in our vocabulary by their similarity to “pleaded”. There are lots of ways to measure the similarity of two vectors. We’ll use the cosine metric:

>>> from numpy import dot
>>> from numpy.linalg import norm

>>> cosine = lambda v1, v2: dot(v1, v2) / (norm(v1) * norm(v2))
>>> words = [w for w in nlp.vocab if w.has_vector]
>>> words.sort(key=lambda w: cosine(w.vector, pleaded.vector))
>>> words.reverse()
>>> print('1-20', ', '.join(w.orth_ for w in words[0:20]))
1-20 pleaded, pled, plead, confessed, interceded, pleads, testified, conspired, motioned, demurred, countersued, remonstrated, begged, apologised, consented, acquiesced, petitioned, quarreled, appealed, pleading
>>> print('50-60', ', '.join(w.orth_ for w in words[50:60]))
50-60 counselled, bragged, backtracked, caucused, refiled, dueled, mused, dissented, yearned, confesses
>>> print('100-110', ', '.join(w.orth_ for w in words[100:110]))
100-110 cabled, ducked, sentenced, perjured, absconded, bargained, overstayed, clerked, confided, sympathizes
>>> print('1000-1010', ', '.join(w.orth_ for w in words[1000:1010]))
1000-1010 scorned, baled, righted, requested, swindled, posited, firebombed, slimed, deferred, sagged
>>> print('50000-50010', ', '.join(w.orth_ for w in words[50000:50010]))
50000-50010, fb, ford, systems, puck, anglers, ik, tabloid, dirty, rims, artists

As you can see, the similarity model that these vectors give us is excellent — we’re still getting meaningful results at 1000 words, off a single prototype! The only problem is that the list really contains two clusters of words: one associated with the legal meaning of “pleaded”, and one for the more general sense. Sorting out these clusters is an area of active research.

A simple work-around is to average the vectors of several words, and use that as our target:

>>> say_verbs = ['pleaded', 'confessed', 'remonstrated', 'begged', 'bragged', 'confided', 'requested']
>>> say_vector = sum(nlp.vocab[verb].vector for verb in say_verbs) / len(say_verbs)
>>> words.sort(key=lambda w: cosine(w.vector * say_vector))
>>> words.reverse()
>>> print('1-20', ', '.join(w.orth_ for w in words[0:20]))
1-20 bragged, remonstrated, enquired, demurred, sighed, mused, intimated, retorted, entreated, motioned, ranted, confided, countersued, gestured, implored, interceded, muttered, marvelled, bickered, despaired
>>> print('50-60', ', '.join(w.orth_ for w in words[50:60]))
50-60 flaunted, quarrelled, ingratiated, vouched, agonized, apologised, lunched, joked, chafed, schemed
>>> print('1000-1010', ', '.join(w.orth_ for w in words[1000:1010]))
1000-1010 hoarded, waded, ensnared, clamoring, abided, deploring, shriveled, endeared, rethought, berate

These definitely look like words that King might scold a writer for attaching adverbs to. Recall that our original adverb highlighting function looked like this:

>>> import spacy.en
>>> from spacy.parts_of_speech import ADV
>>> # Load the pipeline, and call it with some text.
>>> nlp = spacy.en.English()
>>> tokens = nlp("‘Give it back,’ he pleaded abjectly, ‘it’s mine.’",
                          tag=True, parse=False)
>>> print(''.join(tok.string.upper() if tok.pos == ADV else tok.string for tok in tokens))
‘Give it BACK,’ he pleaded ABJECTLY, ‘it’s mine.’

We wanted to refine the logic so that only adverbs modifying evocative verbs of communication, like “pleaded”, were highlighted. We’ve now built a vector that represents that type of word, so now we can highlight adverbs based on subtle logic, honing in on adverbs that seem the most stylistically problematic, given our starting assumptions:

>>> import numpy
>>> from numpy import dot
>>> from numpy.linalg import norm
>>> import spacy.en
>>> from spacy.parts_of_speech import ADV, VERB
>>> cosine = lambda v1, v2: dot(v1, v2) / (norm(v1) * norm(v2))
>>> def is_bad_adverb(token, target_verb, tol):
...   if token.pos != ADV
...     return False
...   elif token.head.pos != VERB:
...     return False
...   elif cosine(token.head.vector, target_verb) < tol:
...     return False
...   else:
...     return True

This example was somewhat contrived — and, truth be told, I’ve never really bought the idea that adverbs were a grave stylistic sin. But hopefully it got the message across: the state-of-the-art NLP technologies are very powerful. spaCy gives you easy and efficient access to them, which lets you build all sorts of useful products and features that were previously impossible.

Independent Evaluation

Independent evaluation by Yahoo! Labs and Emory University, to appear at ACL 2015. Higher is better.

System	Language	Accuracy	Speed
spaCy v0.86	Cython	91.9	13,963
ClearNLP	Java	91.7	10,271
spaCy v0.84	Cython	90.9	13,963
CoreNLP	Java	89.6	8,602
MATE	Java	92.5	550
Turbo	C++	92.4	349
Yara	Java	92.3	340

Joel Tetreault and Amanda Stent (Yahoo! Labs) and Jin-ho Choi (Emory) performed a detailed comparison of the best parsers available. All numbers above are taken from the pre-print they kindly made available to me, except for spaCy v0.86.

I’m particularly grateful to the authors for discussion of their results, which led to the improvement in accuracy between v0.84 and v0.86. A tip from Jin-ho (developer of ClearNLP) was particularly useful.

Detailed Speed Comparison

Per-document processing times. Lower is better.

	Absolute (ms per doc)			Relative (to spaCy)
System	Tokenize	Tag	Parse	Tokenize	Tag	Parse
spaCy	0.2ms	1ms	19ms	1x	1x	1x
CoreNLP	2ms	10ms	49ms	10x	10x	2.6x
ZPar	1ms	8ms	850ms	5x	8x	44.7x
NLTK	4ms	443ms	n/a	20x	443x	n/a

Efficiency is a major concern for NLP applications. It is very common to hear people say that they cannot afford more detailed processing, because their datasets are too large. This is a bad position to be in. If you can’t apply detailed processing, you generally have to cobble together various heuristics. This normally takes a few iterations, and what you come up with will usually be brittle and difficult to reason about.

spaCy’s parser is faster than most taggers, and its tokenizer is fast enough for any workload. And the tokenizer doesn’t just give you a list of strings. A spaCy token is a pointer to a Lexeme struct, from which you can access a wide range of pre-computed features, including embedded word representations.