Search evaluation (ask.chaipat.cc)

The search evaluation is the set of scripts and question files that measure the question engine and stop it from getting worse. It lives in scripts/eval-search.mjs, scripts/plan-snapshot.mjs and scripts/misses.mjs, with its questions in tests/search-eval.json and tests/search-holdout.json.

Two suites, with different jobs

• Regression suite: curated cases that must all pass before anything is published. Each asserts what the engine understood (the kind of question, the task, the operation), where the answer came from, the entities expected among the first results, the action offered and the confidence. A case is added whenever a real question goes wrong, so this suite guards behaviour. It cannot show how well the engine generalises, because it is built from past failures.
• Held-out suite: free-form questions the engine was not tuned against, each with the kind of outcome expected: an answer, a decision not to answer, or a pivot for a technology the graph lacks. node scripts/eval-search.mjs --holdout reports it and never fails the build. It is the yardstick for engine changes, though not a blind one, since the author of the engine wrote it.

The measures are the share of cases with the right outcome, mean reciprocal rank of the expected entity among the answered cases, the time to plan a question, and the confidence gate: how many questions could ever reach a model.

Proving a refactor changed nothing

scripts/plan-snapshot.mjs records what the engine returns for about fourteen hundred questions (every test case, every canned and per-entity question, a few odd inputs) and later compares the engine against the record, down to the order of the results. A cleanup that is meant not to change behaviour is not finished until the comparison is empty.

Learning from visitors

scripts/misses.mjs reads the question log and lists what nothing answered, what had to go to a model and what visitors voted down, each with what the engine would do today and a hint: an alias, a graph fact, a wiki section, a spelling. Nothing changes automatically: a person decides what each miss becomes, and may turn it into a case for the held-out suite.

The results, and their limits, are section 9 of the question engine's paper.

@prefix id: <https://ask.chaipat.cc/id/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix cj: <https://ask.chaipat.cc/ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

id:SiteEval
    a cj:Component ;
    rdfs:label "Search evaluation (ask.chaipat.cc)" ;
    cj:shortLabel "Search evaluation" ;
    cj:componentOf id:ThisSite ;
    cj:sourceFile "scripts/eval-search.mjs" , "scripts/plan-snapshot.mjs" , "scripts/misses.mjs" ;
    cj:implements id:RegressionTesting , id:HeldOutEvaluation , id:MeanReciprocalRank ;
    cj:uses id:NodeJS ;
    cj:description "Measures the question engine with two sets of questions, a regression set that must pass and a held-out set that is only reported; proves a refactor changed nothing by comparing what the engine returns for about fourteen hundred questions; and lists what visitors asked that was answered badly." ;
    cj:article "https://ask.chaipat.cc/wiki/SiteEval"^^xsd:anyURI ;
    cj:implementationDigest "sha256:8c0009b0de0bab31da3fe1eb5a9dd1fb508d0ffa4249449a8b392b77870f3651" .