Wiki full-text search (ask.chaipat.cc)

The wiki full-text search in src/search.js ranks passages of the wiki for a question with BM25. It is plain JavaScript and runs in the visitor's browser, for the wiki's own search box and for the question engine, which can use a selected passage as the evidence for one focused answer.

Every article is split into passages: its lead, and one per paragraph under each heading. Words are lower-cased, common words are dropped, and simple plural and tense endings are stemmed. A passage scores by BM25; an article scores by its best passage, with a bonus when the question names the article itself. A snippet is the best sentence or two of that passage, with the matching words marked.

The index is built by scripts/build-wiki-index.mjs and loaded as one file, data/wiki/search.json. Its existing docs field holds the Markdown passages for BM25. Its separate entities field holds a snapshot of the shared graph-derived catalog: names, aliases, types, descriptions, relationships and available about, live and source resources. This covers every typed entity except the profile, not just entities with written articles.

Passage retrieval answers what an article says; the catalog identifies what an entity is and which actions it offers. src/resources.js builds that catalog from the loaded graph at runtime, which remains authoritative if a generated search snapshot is old. The same module chooses a primary resource from the planner's task without per-project query rules: navigate to a live site, inspect source in a repository, or learn and discover on the local about page. Available alternatives are retained as separate actions. A generated wiki page is enough for an about resource; individual articles need no resource boilerplate.

The snapshot also includes years and relationships from time.ttl, including inferred links. Repository and daily site-date refreshes therefore regenerate GitHub data, then time data, then the wiki graph and search index, and commit the resulting files together. Validation compares the entire entity snapshot with the loaded graph and independently regenerates the exact article passages, so rebuilding only article metadata before time data changes is not enough. The service worker caches both the index and the graph, alongside the resource module, for local offline lookup; a model is not needed to select a destination.

The shared composer in src/answer.js checks that a selected wiki passage actually belongs to the focused article before using it. Its citation links to that article and, when present, its section. Retrieving a related article is not by itself evidence used in the answer, and catalog relationships are not citations. Live/about/source actions remain separate from evidence links.

data/knowledge-state.json hashes every Markdown source, relevant code and schema/graph inputs, as well as generated search bytes. Validation checks those hashes and regenerates expected passage text in read-only mode: both source drift and a stale docs field fail even if the entity catalog is current. Rebuilding refreshes the manifest but cannot establish semantic correctness; changes to ranking, passage extraction or answer selection still require review of this explanation.

@prefix id: <https://ask.chaipat.cc/id/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix cj: <https://ask.chaipat.cc/ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

id:SiteWikiSearch
    a cj:Component ;
    rdfs:label "Wiki full-text search (ask.chaipat.cc)" ;
    cj:shortLabel "Wiki full-text search" ;
    cj:componentOf id:ThisSite ;
    cj:sourceFile "src/search.js" ;
    cj:implements id:BM25 ;
    cj:uses id:JavaScript ;
    cj:description "Splits every wiki article into passages and ranks them for a question with BM25; it finds the passage quoted under an answer and powers the wiki's own search." ;
    cj:article "https://ask.chaipat.cc/wiki/SiteWikiSearch"^^xsd:anyURI ;
    cj:implementationDigest "sha256:789826f7036ff4a54dc599afa17a6edd845aef8f843122f1b0da0ff030178405" .