belayat.uk is a directory of about 828 Nepali businesses and events across the UK, built and run by one person on a 2018-era PHP codebase. This page is a straight account of how I added a natural-language search box to it — "Ask Belayat" — without letting a language model anywhere near the database, plus the eval numbers, the failure modes I found, and the parts of the codebase that had to change first.
The language model's only job is to turn a question into a fixed JSON filter object. It never sees the database, never writes SQL, and never emits a date, a business name or a coordinate. PHP takes that filter, runs an ordinary parameterised query against the existing listings table, and renders the same listing cards the site already had. If the model has an off day, the worst it can do is produce a filter that returns the wrong things or nothing — it cannot produce a business that doesn't exist, because it never generates the answer, only the question's shape.
I considered retrieval-augmented generation and text-to-SQL first and dropped both. At roughly 828 listings, the corpus is small enough that a constrained extraction step is cheaper to run, faster to return, and considerably easier to audit than either — every request produces a JSON object I can log, diff and replay, rather than a generated SQL string or a stitched-together answer I'd have to trust. The trade-off is that Ask Belayat can only do what the schema lets it do. That's deliberate.
Every query the model handles resolves to exactly this shape. Nothing else is a valid response.
A server-side validator sits between the model's output and the query builder: unknown category tokens, out-of-range radii and malformed shapes are rejected before a single row is fetched, not caught after. That validator is doing real work — see the first finding below.
A model asked "what's on this weekend?" will happily produce a concrete calendar date — and, in testing, sometimes the wrong year. So it doesn't get to. The model is only allowed to emit one of a closed set of tokens (this_weekend, next_30_days, and so on); the server resolves that token against the real clock in Europe/London at query time. The model reasons about relative time in words, which is what it's good at. The server does arithmetic on dates, which is what it's good at. Neither one guesses.
54 hand-written test cases, run 2026-08-07, against four candidate models. "Exact match" is the filter object matching the hand-labelled expected output field-for-field; "schema valid" is looser — it only checks the response parses into the schema at all.
| Model | Exact match | Schema valid | p50 latency |
|---|---|---|---|
| Qwen3-30B-A3B-Instruct-2507 | 96% | 100% | 634 ms |
| google/gemma-3-27b-it | 96% | 100% | 706 ms |
| meta-llama/Llama-3.3-70B-Instruct | 91% | 98% | ~1100 ms (29 s p95) |
| Qwen3-32B | 41% | 41% | 8.6 s |
Qwen3-32B is a reasoning model — it emits its chain of thought before the answer, which is exactly what a constrained-extraction task doesn't want. Its low score isn't a mark against the model in general, it's a mismatch between task and model type, and I think it's a more useful result for that reason: it's a reminder to check whether "reasoning" is actually what the task needs before reaching for it.
Ask Belayat runs on the two leading models from this table. 54 cases is enough to rank four models against each other with some confidence; it is not enough to claim a precise accuracy figure in production, and I don't.
My first prompt said, in words, "clamp radius to 1–80km". Asked for "restaurants within 200 miles of Leeds", all four models did the unit conversion correctly and then ignored the clamp, returning 320 km. Stating a rule abstractly wasn't enough. Replacing it with a worked example — "'within 200 miles' becomes 80, NOT 320" — took radius accuracy from 83% to 100% across the board. Separately, the server-side validator caught the 320 km violation 4 times out of 4 in that first run, before the prompt fix existed. That's really the whole argument for having a validator in the first place: the prompt fix made the model behave; the validator is what would have stopped it before the fix existed, and what stops the next thing I haven't thought of.
Asked "what nepali events are on for the 2024 new year", the model came back with date_range: "this_month" and confidence: "high". It had quietly rewritten a stale year as "now" and was confident about it — which is the failure mode that would be easiest to ship without noticing, because the response looks completely normal. Fixed with an explicit rule: any explicit year or calendar date in the question goes into keywords verbatim, and date_range is forced to null rather than guessed.
Both Devanagari script (लन्डनमा नेपाली रेस्टोरेन्ट) and romanised Nepali ("momo kaha paincha", "london ma nepali pasal") scored 100% on the top two models — no term dictionary, no translation step, no special-casing at all. It's the result I was least expecting and the one I'd most like more data on: only 6 of the 54 cases were Nepali-language, which is too small a sample to promise this holds generally. I'm treating it as encouraging, not proven.
No model, across any test case, emitted a category outside the 28-token enum, echoed a coordinate into the near field, or followed an instruction embedded in the query text itself. That's a genuinely clean result, and I'd still rather not depend on it — the validator rejects any category token that isn't a known id regardless of what the model does, so a prompt-injection attempt that a future model does fall for still can't reach the database.
The design goal here was simple: the user always gets results, and the worst-case experience is never worse than the keyword search the site already had before any of this existed.
All three live in a plain MySQL table — there's no Redis or Memcached in this stack, and at this traffic volume a table with an index is genuinely fine. See the honest section below for why I'm not pretending otherwise.
parser_version string, so a prompt change doesn't require walking the cache row by row.Getting the model integration working was, honestly, a smaller job than getting the codebase into a state where it could be called safely. This is a 2018-era PHP site, and some of what I found on the way is worth saying plainly rather than glossing over, because it's the most useful thing a prospective client can take from this page:
None of this is a criticism of whoever wrote the original code under whatever constraints they had. It's just the honest starting point, and a meaningful share of this project's actual effort went into carving stable interfaces out of it before a language model could be pointed at anything.
A search feature is only as good as what it's searching, and the directory has the ordinary mess of five years of manual and semi-automated entry. Some of it is still uncorrected as of this page going up:
None of this is fixed by a better parser. A meaningful share of the work behind this feature was, and continues to be, plain data cleaning — deduplication, recategorisation, closure checks — done by hand against the live directory. I'd rather say that here than let a polished search box imply the underlying data is cleaner than it is.