Repo

Grounded retrieval · eligibility rules · no dependencies

Answers about welfare schemes, with the source attached to every line.

India runs thousands of welfare schemes. The hard part is usually not the money, it is that people who qualify never hear about a scheme, or open the page and cannot work out whether the eligibility rules apply to them.

The obvious thing to do here is hand the question to an LLM. I did not, because a made-up benefit amount does real damage to someone planning around it. So this quotes the scheme documents word for word, puts a citation on every line, and stops when the match is too weak to trust. The retrieval is written by hand: TF-IDF cosine and BM25, combined with reciprocal rank fusion. There is no LangChain, no vector database, and no model call when you ask a question.

Starting Python in your browser…

1Try it

This runs the same Python that is in the repo. The files are fetched from this page and run through Pyodide, so your question stays in your browser.

Some to start with:

2How it finds the right paragraph

Two scoring methods that fail in different ways, combined so neither one runs the show.

  1. Split the schemes into small pieces

    Each scheme's text is cut into overlapping windows of three sentences, with one sentence of overlap. Three sentences is enough for a citation to point at something a person can read, and the overlap stops an answer getting cut in half at a window boundary.

  2. Score every piece twice

    TF-IDF cosine gives a high score when a piece uses the same vocabulary as the question overall. BM25 gives a high score when a rare word from the question shows up in a short piece, with a length correction (k1=1.5, b=0.75). They often disagree, and when they do, usually one of them has latched onto a coincidence.

  3. Combine them by position

    Reciprocal rank fusion adds up 1/(k+rank) from both lists. Since it uses each piece's position in the list and never the raw score, I do not have to calibrate a bounded cosine against an unbounded BM25 number, which was the part I could not get right when I tried combining the scores directly.

  4. Quote the top piece from each scheme

    The answer is the scheme text itself, copied out with an [n] marker after it. Nothing gets reworded on the way, so you can always click through and find the sentence you just read. There is a hook for an LLM to write the answer instead (generate_fn) and it is switched off.

  5. Or stop

    Below a cosine of 0.12 it gives up and points you at the official source. Between 0.12 and 0.22 it also checks that the best piece shares at least two real words with your question, which throws out matches that hang on a single word appearing in both.

3The part I spent the most time on

Getting it to answer was quick. Getting it to shut up at the right moment took much longer.

Try the cricket question above. The retriever still finds a nearest paragraph and still reports a cosine of about 0.17, which is not a small number. A single cut-off either lets that through or throws out real questions that happen to score just as low, because on this dataset the in-scope and out-of-scope score ranges overlap.

That overlap is still there. The two-step check is a patch: a hard floor, then a word-overlap test in the range where I cannot tell the two apart. The proper fix is semantic embeddings, which would compare meaning and stop depending on the exact words used. It is written up in FAILURES.md.

A blank field means unknown

The eligibility checker never fills in a field you left empty. If it does not know your income, the answer comes back as need_more_info and names income as the thing it needs. And where a scheme's rules were pulled out of the document automatically and I have not checked them by hand, the best answer it will give is likely_eligible. It will not say you definitely qualify.

4What it cannot do yet

The code

retriever.py FAILURES.md Repo

Scheme data is compiled from official government portals and every entry links back to where it came from. MIT licensed.