Explosion in 2022: Our Year in Review

It’s been another exciting year at Explosion! We’ve developed a new end-to-end neural coref component for spaCy, improved the speed of our CNN pipelines up to 60%, and published new pre-trained pipelines for Finnish, Korean, Swedish, Croatian and Ukrainian. We’ve also released several updates to Prodigy and introduced new recipes to kickstart annotation with zero- or few-shot learning.

During 2022, we also launched two popular new services – spaCy Tailored Pipelines and spaCy Tailored Analysis. We’ve published several technical blog posts and reports, and created a bunch of new videos covering many tips and tricks to get the most out of our developer tools. We can’t wait to show you what we’re building in 2023 for the next chapters of spaCy and Prodigy, but for now, here’s our look back at 2022. Happy reading!

New spaCy pipeline components

As part of our spaCy v3.3 release in April, we’ve added a trainable lemmatizer to spaCy. It uses edit trees to transform tokens into lemmas and it’s included in the new Finnish, Korean and Swedish pipelines introduced with v3.3, as well as in the new Croatian and Ukrainian pipelines released for spaCy v3.4 in July. We’ve also updated the pipelines for Danish, Dutch, German, Greek, Italian, Lithuanian, Norwegian Bokmål, Polish, Portuguese and Romanian to switch from the lookup or rule-based lemmatizers to the new trainable one.

Furthermore, we’ve implemented a new end-to-end neural coreference resolution component in spacy-experimental’s v0.6.0 release. The release includes an experimental English coref pipeline and a sample project that shows how to train a coref model for spaCy. You can read all about this new coref component in our blog post by Ákos, Paul and team that outlines why you want to do coreference resolution in the first place and explains some of the crucial architecture choices of our end-to-end neural system in detail. Finally, Edi recorded a video on our coref component showing how to train a coreference resolution model with spaCy projects and then applies the trained pipeline to resolve references in a text. Use these resources to jump-start your experiments with coref, and let us know how you go on the discussion forum!

Finally, we’ve spent some time running various experiments and implementing extensions to our new SpanCategorizer component. The spancat is a spaCy component that answers the need to handle arbitrary and overlapping spans, which can be used for long phrases, non-named entities or overlapping annotations. We added some useful span suggesters to spacy-experimental v0.5.0 that identify candidate spans by inspecting annotations from the tagger and parser, and then marking relevant subtrees, noun chunks, or sentences. Edi, Lj and team have written a comprehensive blog post covering full details of the spancat implementation as well as an architecture case study on nested NER. In his most recent video, Edi shows how to use Prodigy for spaCy’s spancat component, annotating food recipes and sharing best practices around annotation consistency and efficiency.

Performance improvements in our open-source stack

We’ve been focusing heavily on speed improvements across our open-source stack for two years now, including spaCy and Thinc. We fixed a lot of the low-hanging fruit in 2021, improving transformer training performance by up to 62%. We’ve achieved further improvements in 2022 by systematically profiling training and inference and eliminating bottlenecks where we could. There were too many improvements to summarize here, so we will highlight three changes:

1. The Thinc Softmax layer is used by many models to compute a probability distribution over classes. This function is quite expensive due to its use of the exponentiation function. During inference, we usually do not care about the actual class probabilities but rather what the most probable class is. Since softmax is a monotonic function, we can find the most probable class from the raw inputs to the softmax function (the so-called logits). In spaCy v3.3, we started using logits during inference, which resulted in speedups of 27% when using a tagging + parsing pipeline.
2. The transition-based parser extracts features for the transition model to predict the next transition. One function that is used in feature extraction looks up the n-th most recent left-arc of a head. In order to do so, it would first extract all arcs with that particular head from a table of all left-arcs. Since the number of left-arcs correlates with the document length, doing this for each transition unfortunately degraded the complexity of the parser to quadratic-time. In spaCy v3.3, we rewrote this function to perform the lookup in constant-time, restoring the parser’s overall complexity to linear time again. This resulted in large speedups on long documents.
3. One of the operations involved in the training of a pipeline component is the calculation of the loss between the model’s predictions and the gold-standard labels, which requires computing the alignment between the two. Originally, the alignment function manually iterated through arrays using a for-loop and compared the entries individually. In spaCy v3.4, we vectorized those operations which increased GPU throughput and reduced training time by 20%.

Thanks to the aforementioned changes and a myriad other, smaller optimizations, we’ve been able to squeeze out significant improvements in both inference and training performance. In the tables below, we compare the inference and training performance of spaCy on January 1, 2022 and January 1, 2023 for a German pipeline with the tagger, morphologizer, parser and attribute ruler components. The results show improvements across the board, but are most visible in pipelines that are not dominated by matrix multiplication.

Inference performance on Ryzen 5950X/GeForce RTX3090

Pipeline	Device	January 2022 (words/s)	January 2023 (words/s)	Delta
Convolution	CPU	25,421	25,573	+0.6%
Convolution	GPU	96,291	121,623	+26.3%
Transformer	CPU	1,743	1,779	+2.0%
Transformer	GPU	20,381	20,297	-0.4%

Training performance on Ryzen 5950X/GeForce RTX3090

Pipeline	Device	January 2022 (words/s)	January 2023 (words/s)	Delta
Convolution	CPU	5,139	6,359	+23.7%
Convolution	GPU	4,667	5,139	+10.0%
Transformer	GPU	3,327	3,575	+7.5%

We also made two large optimizations that primarily benefit Apple Silicon Macs. In 2021, we released thinc-apple-ops. With this add-on package, Thinc uses Apple’s Accelerate framework for matrix multiplication. Accelerate uses special matrix multiplication units (AMX) on Apple Silicon Macs, resulting in large speedups. However, spaCy’s dependency parser did not use Thinc for matrix multiplication in low-level Cython code. The first optimization was to define a C BLAS interface in Thinc and use this in the dependency parser to leverage the AMX units. This leads to large improvements in training and inference speeds as shown in the tables below.

The second optimization was to leverage the support for Metal Performance Shaders that was added to PyTorch to speed up transformer models. Madeesh and Daniël have written a blog post about fast transformer inference using Metal Performance Shaders. The performance impact can also be seen in the results below.

Inference performance on M1 Max

Pipeline	Device	January 2022 (words/s)	January 2023 (words/s)	Delta
Convolution	CPU	35,818	57,376	+60.1%
Transformer	CPU	1,883	1,887	0.0%
Transformer	GPU	See CPU	7,660	+406.9%

Training performance on M1 Max

Pipeline	Device	January 2022 (words/s)	January 2023 (words/s)	Delta
Convolution	CPU	7,593	9975	+31.4%

All in all, it has been a great year for performance! Nevertheless, we have more improvements in the works - particularly with respect to transformer models - that we hope to show you in the following months.

Prodigy updates

We released Prodigy v1.11.7 and Prodigy v1.11.8. These releases include various bug fixes, usability improvements and extended support to the latest spaCy versions, as well as many other small improvements.

Further, we’ve been working on new Prodigy workflows that use the OpenAI API to kickstart your annotations, via zero- or few-shot learning. We published the first recipe, for NER annotation, at the end of December. Keep an eye on this repo as more exciting recipes will be published soon!

Launching our consultancy offerings

In 2022, we launched two brand new consulting services! February saw the launch of spaCy Tailored Pipelines, where we offer custom-made solutions by spaCy’s core developers for your NLP problems. By the summer we had already engaged with several companies on a variety of interesting use cases, including Patent Bots’ legal information extraction pipeline. It now handles training, packaging and deployment in a spaCy project structure that is easy to maintain and update in the future.

In November, we followed up with the launch of our second new service: spaCy Tailored Analysis. People often ask us for help with problem solving, strategy and analysis for their applied NLP projects, so we designed this new service to help with exactly these types of problems.

Open-source stack

In August, we released the config system used by spaCy and Thinc as its own lightweight package: confection! Confection, our battle-tested config system for Python, can now easily be included in any Python project without having to install Thinc.

We’ve also added support for spaCy v3.4 for English, German and Polish in the v1.3.0 release of the Coreferee library. Holmes, an information extraction component based on predicate logic, was also updated to support spaCy v3.4 in its v4.1.0 release.

User resources and documentation

We’ve worked hard on creating more resources that explain spaCy’s implementation and architecture choices in further detail. On top of the content produced for coref and spancat, Adriane has written an interesting blog post explaining floret, which combines fastText and Bloom embeddings to create compact vector tables with both word and subword information and enables vectors that are up to 10× smaller than traditional word vectors. Additionally, Lj, Ákos et al. published a technical report that benchmarks spaCy’s hashing trick on various NER datasets in different scenarios. Finally, this LinkedIn thread by Vincent explains you all you need to know about spaCy’s Vocab object and its vectors.

Only just getting started with spaCy or Prodigy? Our ever-popular “Advanced NLP with spaCy” course has got you covered, and is now available en français on top of our current languages: English, German, Spanish, Portuguese, Japanese and Chinese. We’ve also created a spaCy cheat sheet, packed with great features and practical tips so you can create projects at lightning speed, and we revamped Ines’ flowchart containing our best practices for annotating and training Named Entity Recognition models with Prodigy. The PDF version includes clickable links for context and additional information.

Already a pro? You might be pretty interested in The Guardian case study report that Ryan and team wrote. In order to modularize content for reuse, The Guardian’s data science team developed a spaCy-Prodigy NER workflow for quote extraction. We talked with The Guardian’s lead data scientist Anna Vissens about the project for a fascinating blog post. And on the topic of expert content, our machine learning engineer Lj shows how to integrate HuggingFace’s LayoutLMv3 model with Prodigy to tackle the challenge of extracting information from PDFs.

Videos

We’ve expanded our YouTube channel with two new playlists: spaCy Shorts and Prodigy Shorts. As part of the spaCy Shorts series, Vincent walks you through various quick lessons on how to speed up your pipeline execution via nlp.pipe, how to leverage linguistic features in a rule-based approach, and much more. The bite-sized videos in the Prodigy Shorts playlist demonstrate how to configure the Prodigy UI for efficient annotations, and how to exploit Prodigy’s core scriptability design.

Interested in more nitty gritty details? In one of his first videos with Explosion, Vincent explains how to use Prodigy to train a named entity recognition model from scratch by taking advantage of semi-automatic annotation and modern transfer learning techniques. On the topic of efficient labeling, this recent Prodigy video shows how you can use a bulk labeling technique to prepare data for Prodigy and illustrates that a pre-trained language model can help you annotate data. Finally, this Prodigy video shows how you might be able to improve the annotation experience by leveraging sense2vec to pre-fill named entities.

Talks

We are always excited to talk about our vision on implementing developer tools, general design choices or new features that we released. In January, Ines appeared on ZenML’s podcast Pipeline Conversations and talked about creating tools that spark joy. She also gave the keynote at the New Languages for NLP conference at Princeton in May. Her talk covered the challenges for non-English NLP and how spaCy allows you to develop advanced NLP pipelines, including for typologically diverse languages. In June, she presented a nice recap of spaCy’s changes over time on Deepak John Reji’s D4 Data Podcast.

Over at the Data-aware Distributed Computing (DADC) conference in July, Damian and Magda gave a talk on collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop. Victoria and Damian also both gave talks at PyData Global in the beginning of December.

If you were ever curious about what some of us get up to at Explosion, as of December we’ve added an events page to our website where you can see upcoming and past talks from us. If you want to meet us in person and learn about our tools, maybe grab some stickers, check it out!

With the community and the team continuing to grow, we look forward to making 2023 even better. Thanks for all your support!