Jev for Search Intent Classification: 86% Right Across Five Sites, and Where It Fails
Search intent labels decide which keywords get turned into content, and so it matters a great deal whether those labels are right. We tested Jev - a small AI model from TypeSafe - against a blind reader on five sites that it had never seen and it matched the reader on about 86% of queries. On a separate set of keywords a paid intent classifier matched the same kind of reader on about 48%.
Key takeaways
- Jev matched a blind reader on 86.7% of queries across five B2B sites it had never seen (500 labelled queries, October 2026).
- Counting either of the reader's two acceptable intents it matched on 94.6%. Many short queries have two fair readings.
- A paid intent classifier matched on about 48% of a 200-keyword sample, where Jev matched on about 74%.
- A more elaborate four-question version did no better on unseen sites and cost four times as much.
- Jev is weakest on transactional queries, one-word queries and brand names that are everyday phrases.
About the data: the scores in this post come from 16,228 keywords and queries belonging to five client companies. They are anonymised - described by industry, never named, and no client query is quoted. The example queries shown are from our own site, tenpointlabs.com (Search Console web queries, 9 June 2025 to 9 October 2026, a sample of our traffic and not all of it), which is one part of the data used across this series.
What is search intent classification?
Search intent classification sorts each query by what the person searching wants to do. The four labels in general use are:
- Informational: learn or understand something.
- Commercial: research or compare products before choosing.
- Transactional: do something now, such as buy, sign up or use a tool.
- Navigational: reach one specific company, product or page.
SEO tools attach these labels automatically and whole content plans are then built on top of them. A keyword that is marked commercial gets a comparison page while one that is marked informational gets a guide. Intent is one of the six steps in our guide to keyword research for B2B SEO.
What did we find?
Table 1. Share of queries where each method matched a blind reader's label
| Test | Rows labelled | Paid classifier | Jev, one question | Jev, four questions |
|---|---|---|---|---|
| First set: one healthcare software company's keywords | 200 | about 48% | about 74% | about 86% |
| Five unseen sites, all together | 500 | not run | 86.7% | 85.1% |
On the first set the four-question version looked 12 points better than the plain one. However on five new sites that lead vanished entirely. I had already written a sentence saying that the redesign "helped" and the hold-out test is the reason that sentence never shipped.
Table 2. Jev's one-question design on five unseen sites, by industry
| Site | Matched the reader | Matched either acceptable intent |
|---|---|---|
| Retail software | 81.7% | 94.4% |
| Healthcare software | 80.9% | 87.1% |
| Cybersecurity software | 89.9% | 96.8% |
| Inventory software | 94.3% | 98.8% |
| B2B AI platform | 86.6% | 95.7% |
Why did the paid classifier score so low?
On the keywords where the classifier and Jev disagreed a blind reader sided with Jev on 86 of 120 and with the classifier on only 17. Two patterns account for most of that gap:
- The classifier labelled topic keywords as commercial or transactional. A keyword about missed-appointment rates was "commercial", where both Jev and the reader saw someone trying to learn.
- It labelled company names as informational, where both Jev and the reader saw someone trying to reach that company.
Before the reader's labels came in the only number we had was that Jev and the classifier agreed on 54.9% of keywords, and on its own that looked very much like Jev failing. Agreement between two tools can tell you that they differ, but it cannot tell you which of the two to believe.
What does this mean for your content plan?
If the intent labels in your keyword tool are wrong about as often as they are right then at least some of your content priorities are resting on noise.
- Check a sample before you trust the column. Fifty keywords is enough to see whether your tool's labels match your own reading.
- Expect two intents on short keywords. "domain name rating" could be someone asking what the metric means or someone looking for a checker.
- Decide your own rule for brand queries. We count a brand name plus a product word as navigational, and your team should write its own rule down because tools and people both differ on it.
How can you check your own tool in 15 minutes?
- Export 50 keywords from your SEO tool with its intent label.
- Hide the label column and write your own label beside each keyword.
- Unhide and count the matches.
- Read the mismatches. Sort them into the tool being wrong, you being unsure, and keywords with two fair readings.
If the match rate comes in below about 70% then I would stop using that column to set priorities.
The question we sent to Jev
Each query is sent to Jev in a single request and the query itself is the only input:
{
"type": "choice",
"instructions": "What is the person who searched `query` trying to do?",
"criteria": {
"informational": "Learn or understand something: a definition, an explanation, how something works, how to do it.",
"commercial": "Research or compare products, services, tools or providers before choosing one.",
"transactional": "Do something now: buy, sign up, book, download, or use a tool or calculator.",
"navigational": "Reach one specific named company, website or page."
}
}
Jev returns the label it has chosen along with a probability for each of the four options. Here is what that looks like on four queries from our own site:
| Query | Jev's label | Probability |
|---|---|---|
| saas seo agency | commercial | 0.99 |
| domain rating checker | transactional | 0.96 |
| b2b seo | informational | 0.92 |
| domain name rating | informational | 0.50, with transactional at 0.41 |
Where does Jev fail?
- Transactional queries. The two Jev designs agreed with each other on only about half of the queries either called transactional.
- One- and two-word queries. The four-question version was unsure on 26% of them.
- Brands that are everyday phrases. On one site the company's own name is a common two-word phrase and neither design read it as a brand search.
- Close calls change between runs. Sending the same 300 keywords twice changed 6 labels, all with a top probability of 0.54 or lower.
- Option order has a small effect. Reversing the four options changed 16 of 300 labels, none among those at 0.8 or above.
Before we tested Jev our own tool labelled intent with trigger-word rules, and Jev vs regex shows how much those rules left blank.
How we tested it
- First set: 1,326 keywords from a healthcare software company's research, each already labelled by a paid intent classifier.
- Unseen sites: a random sample of up to 3,000 Search Console queries from each of five B2B sites, April to September 2026, 14,902 in all. The Jev designs were fixed before these were run.
- Labels: a larger AI model labelled 200 keywords from the first set and 100 queries per unseen site, seeing only the text. I labelled 30 myself and agreed with it on 21 of 28, with 2 queries that had no real intent.
- Sampling: each sample over-represents rows where methods disagreed, and the percentages above are weighted back to the full set.
- Model and cost:
jev-1.13.0, about 420 input tokens per query for one question and 4.1 times that for four.
Limits. One reader labelled each sample and was unsure on about half the rows. The paid classifier was scored on one company's keywords only, which is why we don't name it.
FAQs
How accurate is search intent classification?
It depends on the tool. In our tests Jev matched a blind reader on about 86% of queries across five sites and a paid classifier matched on about 48% of one keyword set.
What are the four types of search intent?
Informational, commercial, transactional and navigational. Many short queries fit two of them.
Can Jev classify search intent?
Yes. A single four-way question was as good as a more elaborate design on unseen sites and cost a quarter as much.
Why do two tools give different intent labels for the same keyword?
They are trained on different signals and use different definitions, especially for brand names and for topics that are also product categories.
Is a brand name plus a product word navigational or commercial?
People disagree and so do tools. We treat it as navigational. What matters is that your team picks one rule and applies it.
See how AI engines answer your buyers
Intent labels tell you what a searcher wants. Our Free AI Visibility Audit shows what AI engines tell them and whether your company is named. As more of these labels are produced by small models it will be worth keeping a close eye on who is checking them.
This post is part of a series: Jev for SEO: what worked, what failed, and what we measured.
