

We’ve talked to Trevor Beers, AI Lead at Popp, about how their team used Ellf for agentic NLP development to assess and replace LLM calls behind a core part of their product with System One models, cutting token costs by 76% and latency by more than 7× while making the system more accurate on every decision.
Popp builds AI tools for recruiters. Their conversational AI runs the initial screening of candidates over email, SMS and WhatsApp: recruiters configure the questions they want to ask and upload the job description, and Popp has the conversations, collects documents and hands over to the recruiter whenever a candidate asks for a person or has a question it can’t answer. Popp has had over 1.2 million conversations with candidates so far, saving recruiters more than 20,000 days of work.
Recruitment AI that keeps humans in the loop
While many AI recruitment solutions today try to replace the recruiter entirely, Popp takes a different approach: give recruiters tools for the repetitive parts of the job so they can spend more time on the human parts.
We’re building for the people doing the actual hands-on work, so they can reach more candidates and spend their time where it matters. Our conversational screening is audited every month by an independent auditor, Warden, to make sure our system treats every candidate fairly.
— Trevor Beers, AI Lead
Under the hood, each conversation is driven by a deterministic flowchart, and every message a candidate sends passes through a set of text classification decisions. Before the conversation can move on, Popp’s conversation engine runs a series of early-exit checks: does the candidate want to opt out? Are they not interested in the role? Do they want to talk to a human? Are they asking to be contacted later? Is the message negative, or even abusive? Each of these is a classic NLP task with a direct effect on what the candidate experiences next.

NLP everywhere, and an LLM for everything
Like many young companies moving fast, Popp had implemented all of these steps by prompting general-purpose LLMs from OpenAI. That got the product off the ground, but every call is relatively expensive and slow, replies sometimes fail to parse, and there’s no usable confidence score or clear picture of how well each step actually performs.
Before this project, everything was a generative LLM call. I knew there had to be a better way – but without a clear picture of how our pipelines were performing, it felt like shooting in the dark.
— Trevor Beers, AI Lead
Accuracy matters a lot here: there’s always a real person on the receiving end of each decision, and Popp’s customers review conversations closely, so every wrong decision shows up as a bug report. But reliable accuracy numbers are hard to produce without a proper evaluation workflow in place.
Evaluate and replace: Ellf and Jev
Ellf gives coding assistants like Claude Code or Codex superpowers for solving NLP problems and a platform to execute data-centric projects, automated annotation, evaluation and more. It also comes with project templates to set up end-to-end workflows for common tasks with a single prompt. To process their data, Popp plugged in a private cluster on their own cloud account, so no sensitive information like candidate messages ever had to leave their servers.

Instead of starting with a new model, the project started with the question every NLP project should start with: how well is the current system actually doing, and where can it be improved? Each of the early-exit decisions was treated as a separate component that could be evaluated on its own and replaced individually if a better, faster or cheaper alternative existed.
One of those alternatives, and one Trevor already had his eye on, was Jev by TypeSafe. Jev is a decision model: instead of generating text, it answers typed questions, like yes/no or a choice between several labels, and returns a probability for each answer. But Jev isn’t a drop-in replacement. Getting the best results means writing good instructions and label criteria, choosing the right thresholds and encoding the business logic that no model can infer on its own.
The end-to-end workflow
The workflow Trevor developed with Ellf is designed to be repeatable for every component and to keep running as new data comes in:
- Pull production data from Popp’s logs in Logfire.
- Annotate 200–300 messages per task to create a small, high-quality evaluation set.
- Optimize the Jev configs: Ellf tunes the instructions, label criteria, thresholds and rules for each decision to achieve the highest possible precision and recall on the annotated data.
- Shadow release: The new Jev stack runs in the background on 25% of live messages, next to the production LLM calls. The answers are only logged and the reply never waits for Jev.
- Annotate disagreements between the LLM and Jev after a few days. When the two agree, Jev can’t be worse, so only the disagreements need to be reviewed to see which model is right. Then have Ellf re-optimize the configs where needed.
- Release, replacing the LLM calls with Jev.
Because the workflow is now available as a project template, setting up the production version of the development workflow with its tasks and actions in the platform only took a single prompt.
Ellf made it really easy to set up the multi-step workflow straight from Claude Code. I especially love the templates: I told it to import the LLM-to-Jev template and customize it for our use case – and it did it all for me!
— Trevor Beers, AI Lead
Results
Across the early-exit decisions, replacing the LLM calls with Jev cut the token costs by about 76%. The median response time for this part of the pipeline went down from about 1.5 seconds to 0.2 seconds, and 99% of requests now complete within 0.4 seconds, compared to 3.8 seconds before, all while improving the F-score on every single decision.
- token cost savings per year
- 76%
- faster response times
- 7×
- max. accuracy improvement
- +23%
| OpenAI (before) | Jev (after) | |||||
|---|---|---|---|---|---|---|
| Decision | F1 | Latency1 | Cost2 | F1 | Latency1 | Cost2 |
| Opt-out | 0.87 | 1.54s | $0.00079 | 0.91 | 0.20s | $0.00016 |
| Not interested | 0.79 | 1.26s | $0.00150 | 0.84 | 0.20s | $0.00016 |
| Wants human | 0.74 | 1.54s | $0.00079 | 0.92 | 0.20s | $0.00016 |
| Delay | 0.64 | 1.54s | $0.00079 | 0.87 | 0.20s | $0.00016 |
| Negative3 | 0.50 | 1.03s | $0.00007 | 0.88 | 0.20s | $0.00016 |
| Abusive3 | – | 1.03s | $0.00007 | 1.00 | 0.20s | $0.00016 |
- Median latency: 50% of requests are processed within this time.
- Cost per classification in USD.
- Sentiment classification was redesigned as part of the project: the previous OpenAI pipeline predicted 12 fine-grained sentiment labels, while Jev only predicts the two categories that matter, negative and abusive, so the results aren’t directly comparable.
For the sentiment labels, Jev is actually slightly more expensive per call than the small LLM used previously, but this is more than made up for by the savings on the other decisions, and the redesigned labels are now accurate enough to act on, for example to flag negative messages for review.
Human annotation is the critical part
The project initially started out with a different strategy: treating the existing LLM as the reference, measuring how often Jev agreed with it, and only asking a human to judge the disagreements. This quickly showed that most disagreements came from vague category definitions rather than model errors. Is “Not currently available for travel assignments” a negative message, or just a polite decline? Is “Unsubscribe” negative? These are business decisions, not modeling problems, and a human needs to make them.

The final workflow puts human annotation front and center. With Ellf’s built-in annotation powered by Prodigy, annotating 200–300 messages per task took Trevor roughly one hour per component. Decisions made during annotation can go straight back into Jev’s instructions as rules: for example, a short “no”, a polite decline or a plain opt-out isn’t negative by itself unless the candidate also sounds unhappy.
One of the biggest takeaways for me was how important human annotation is. We’re scrapping all our LLM-as-a-judge workflows and going back to pulling production data every two weeks and annotating it. It’s only half an hour of my time every two weeks, and totally worth it.
— Trevor Beers, AI Lead
This also reflects a broader shift in how the team approaches AI engineering: the critical part of the process is no longer writing prompts, but defining what the right answer should be, and checking that the system produces it.
Future plans
With the workflow in place, Popp now has a robust and repeatable process to evaluate their system as new data comes in and find out where to improve it. The team is rolling out the new conversational pipelines to production, and is setting up fortnightly reviews where engineers take turns annotating a sample of production data in Ellf to make sure everything stays on track.
Next up is Popp’s CV scoring system, the slowest and most expensive part of the product. Reproducing the same process there using the template is expected to lead to even bigger cost savings and improvements.
As the product matures, the focus is shifting towards optimizing both cost and performance. For the simpler decisions, this could mean going even further and training small custom models. With the evaluation data already in place, it’ll be easy to tell whether a custom model is worth it, and all training experiments can run in Ellf as well.
As a young company, we like trying out new things and getting in early, and working with Ellf has been a great experience. We had real influence on the product along the way, and we came out of it with a process we can now reuse across the rest of our product.
— Trevor Beers, AI Lead
Resources
- Ellf: Our new virtual assistant and platform for agentic NLP development
- Introducing Ellf: The missing piece for building NLP with coding assistants
- Replacing an LLM Classifier with Jev: Project template for the workflow described in this post
- Project Templates: End-to-end workflows for common NLP tasks
- Jev by TypeSafe: System One models for fast, typed decisions
- A practical guide to human-in-the-loop distillation: How to distill LLMs into smaller, faster, private and more accurate components
- Popp: AI-assisted tools for recruiters
- Case Studies: Our other real-world case studies from industry



