How to Write Jev Questions for SEO: What Changed Our Results and What Didn't
Jev answers the question that you wrote, and across four SEO tests small changes in wording moved our results more than anything else we tried. Asking about a name in the form it took in the text lifted one score from 0.07 to 0.97. However splitting an intent question into four did nothing at all on new data and cost four times as much. These are the seven lessons we took away.
Key takeaways
- Ask about what is in the text. A question about who wrote a query got 5 confident answers in 7,157 and questions about its visible features got far more.
- Use the wording that appears in the text. Asking about a stored name when the passage used a short form lost about 6% of real brand mentions.
- Give every option a boundary. Adding "not for" lines to 16 keyword labels cut one over-used label from 203 keywords to 65.
- A redesign that helps on one dataset may not help on the next, as ours showed on search intent.
- Score against blind labels. Agreement with another model misled us in both directions.
About the data: the lessons draw on four tests. One used 2,944 queries from our own site, tenpointlabs.com, and those are the only real queries quoted. These are the queries Search Console reports for web search between 9 June 2025 and 9 October 2026. They are a sample of our traffic and not all of it, because Search Console leaves out anonymised queries, and no GA4 data is used. The other three used client and prospect data, which is anonymised - described by industry, never named, and quoted only with the names changed.
What is a Jev question?
Jev is a model from TypeSafe that takes some text - which it calls the state - along with one or more typed questions. A question has instructions and criteria and comes in three types:
- Noul: a yes-or-no question, answered with a probability between 0 and 1.
- Choice: one option from a list, with a probability for each.
- Score: a level on a scale you define.
This is one of the technical posts in the series and is written for whoever builds the pipeline. The lessons are likely to apply to any small classification model, and the numbers are from jev-1.13.0 in October 2026.
1. Does the text contain the answer?
Our first question for spotting AI-generated search queries was "Was this query written by an AI assistant?" Authorship is simply absent from a query string and so Jev's answers clustered near 0.5, with only 5 of 7,157 queries reaching 0.8.
We replaced that question with eight others about things that anyone can see, such as "Is query written as a complete, grammatical sentence or question?" and "Does query read like a filled-in template?" Two of the eight were decisive on more than 99% of queries, and code then combined them with click and position data that Jev never saw. The full test is written up in Jev for query fan-out.
The lesson: if a careful person could not answer from the text alone then either change the question or put the missing evidence into the state.
2. Are you asking about the words that are there?
Our brand-mention question named the brand as we store it, so when a passage used a short form, Jev said no.
| Question | Probability |
|---|---|
| Is "Ledgerline AI" used as the name of the company called Ledgerline AI? | 0.07 |
| Does "Ledgerline" refer to the company described as Ledgerline AI, also written as Ledgerline? | 0.97 |
The passage - altered from a real one - read "Ledgerline can help coordinate the month-end close…". Across 2,925 real name checks Jev said yes to 98.3% of mentions written in the stored form and to 10.7% of those written in a short form. The full test is in Jev for brand mention tracking.
The lesson: build the question from the string your code found and pass the canonical name as context.
3. Do your instructions and your examples agree?
For brand queries we asked "Does query name the company Tenpoint Labs?" and gave "tenpoint" as a spelling that counts. Jev then scored the query "tenpoint" at 0.49.
TypeSafe's notes list conflicting instructions and criteria as a known weak spot in the model. Our instruction said "the company Tenpoint Labs" while our example allowed a bare short form, and the model split the difference between the two.
The lesson: read the instruction and each criterion together as though they were one sentence, and if they could be answered differently then rewrite them until they can't.
4. Does each option say what it is not for?
Our keyword-relevance test sorted 2,804 keywords into 16 labels using one-line definitions that had been written for a larger model. Jev gave a voice-agent label to 203 keywords where the larger model had used it for 64.
We rewrote each option to say what it covers, what belongs to a neighbouring label instead, and to give three invented examples. We also rewrote the company description so that it didn't favour any one label.
Table 1. Keyword relevance before and after adding boundaries (2,804 keywords, one healthcare software company)
| One-line definitions | With boundaries and examples | |
|---|---|---|
| Keywords given the voice-agent label (larger model: 64) | 203 | 65 |
| Share of keywords answered at 0.8 or higher | 50.3% | 64.9% |
| Estimated match with a blind reader | about 74% | about 87% |
| Input tokens per keyword | 900 | 2,440 |
The larger model matched the same reader on about 83%. I would call that level, since the reader gave a second acceptable label on half the keywords and when I labelled 24 of them myself I matched the reader on only 14.
The lesson: overlapping options need explicit boundaries, and those boundaries cost tokens.
5. Will the new wording hold on new data?
For search intent we replaced a single four-way choice with four separate yes-or-no questions, each with its own boundaries and examples. On the first dataset the new version looked about 12 points better against a blind reader. However on five sites that it had never seen it scored 85.1% against 86.7% for the plain version. The full results are in Jev for search intent classification.
The lesson: fix the design first and then test it on data it hasn't seen. We now write the request down before the full run and record its hash.
6. Have you checked option order and repeat runs?
- Order. Reversing the options changed 16 of 300 intent labels and 35 of 400 relevance labels. Almost all were on rows below 0.8. With boundaries added, the relevance figure fell to 19 of 400.
- Repeats. Sending identical requests twice changed up to 3% of final labels. The probabilities themselves moved on about a quarter of requests, by 0.15 at most.
The lesson: treat answers below 0.8 as provisional and send them to a second opinion.
7. What are you scoring against?
Jev agreed with a paid intent classifier on 54.9% of keywords, which on its own looked like failure. A blind reader then put Jev at about 74% and the classifier at about 48%. The same thing happened in the other direction - the boundary rewrite in lesson 4 moved agreement with the larger model by only 3 points while the reader saw about 13.
The lesson: agreement with a model or a rule is a measure of similarity. To measure accuracy you need someone to label a sample without seeing any method's answer, and you need to say how far a second person agrees with them.
A checklist before you run anything
- Can a person answer this from the state alone?
- Does the question use the words that appear in the text?
- Is each question one condition, with no "or" joining different things?
- Do the instructions and every criterion point the same way?
- Does each option say what belongs to its neighbour?
- Is the state free of text the question doesn't need?
- Is there a blind-labelled sample to score against?
- Is the design written down before the full run?
FAQs
How do you write a good Jev question?
Ask one narrow thing that the text can answer, use the words that appear in the text, and give each option a clear boundary. Then test on data the design hasn't seen.
Should I split one Jev question into several?
Sometimes. Splitting helped when the single question asked for something the text didn't contain, and on search intent it changed nothing while costing four times the tokens.
Does option order affect Jev's answers?
Reversing the options changed 5% to 9% of labels in our tests, nearly all on low-confidence rows.
Does Jev give the same answer every time?
Final labels changed on up to 3% of identical requests in our tests, always on close calls.
How do I measure whether a Jev question is accurate?
Have a person or a stronger model label a sample without seeing Jev's answers, and compare. Agreement with an existing tool is not enough.
See where your brand stands in AI answers
Lesson 2 came from our own AI visibility product. The Free AI Visibility Audit runs your buyers' questions through the major AI engines and shows who gets named. These models are improving quickly and it will be worth watching which of these lessons still hold in the versions to come.
This post is part of a series: Jev for SEO: what worked, what failed, and what we measured.
