Evaluating retrieval quality without a labelled set

The standard advice for evaluating a retrieval system assumes a labelled set: questions paired with the documents that should have been returned. Most enterprises do not have one, and building it properly is a months-long annotation project nobody has budget for.

There is a reasonable path that does not require one. Start with synthetic questions generated from the corpus itself: take a passage, generate a question it answers, and check whether retrieval returns that passage. This is not a substitute for real user questions, but it catches chunking and embedding problems quickly and costs almost nothing.

Layer on human judgement where it is cheapest. Ask three domain experts to rate fifty real answers on a simple scale, and use their disagreements to find the queries that actually matter. Fifty carefully chosen examples tell you more than a thousand generated ones.

Finally, instrument the live system. Log which retrieved passages the generation step actually cited, and treat a passage that is retrieved but never cited as a signal that ranking is off. Production traffic is the labelled set you did not have to build.