Model fallback (ask.chaipat.cc)

The model fallback is the part of ask.chaipat.cc that could ask a hosted language model about a question the question engine is unsure of. It lives in api/ask.js, with lib/llm.js and lib/verify.js, and it is switched off: nothing is sent to a model unless the setting LLM_ENABLED is true and a provider key is set, and a key on its own enables nothing. With the default setting, no question leaves the server.

Why it exists, and why it is off

Every answer the engine gives carries a confidence. An authored or graph answer, or an answer about something the question names, is trusted; an answer found only because some words overlap is a guess, and on a portfolio a wrong answer is worse than none. The fallback is the design for those guesses: it gives a model only the passages the engine retrieved, lets it query the graph, and shows its answer only if the answer survives checking. This is a retrieval-augmented generation design, with the checking added.

It is off for two reasons. It has been run against a simulated model in tests, but never evaluated against a live one, so nobody can say yet whether it helps. And while it is off, the question a visitor types stays on the server.

How it would work

1. Evidence. The five best-matching wiki passages, the plan's own passage and any definition are packed into the prompt, with the graph's vocabulary but not its list of every entity.
2. The model, through one function that speaks to Groq or to Anthropic, either failing over to the other. It has two tools, a SPARQL query and a wiki search, and at most three rounds.
3. Links are removed. Any address the server did not supply is stripped from the answer.
4. Checks. The answer is discarded, and the visitor sees "I couldn't find that in the graph or the wiki", if it names a known entity, a number, or a proper noun or technical term that no evidence contains, if it contains a phone number or a birth date, or if it cites no source.

The checks are conservative on purpose, so they can discard a good answer. They do not prove that an answer is true: a false claim made only of ordinary words would pass.

Cost

A request is about three thousand input tokens, after the first prompt, at eight thousand, was found to exceed a free plan's limit on a single request. A plan that allows seven thousand input tokens a minute answers about two questions a minute, and a question where the model calls a tool sends two requests. A visitor who hits the limit keeps the engine's own answer.

The design, with pseudocode and a flowchart, is section 8 of the question engine's paper.

@prefix id: <https://ask.chaipat.cc/id/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix cj: <https://ask.chaipat.cc/ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

id:SiteFallback
    a cj:Component ;
    rdfs:label "Model fallback (ask.chaipat.cc)" ;
    cj:shortLabel "Model fallback" ;
    skos:altLabel "fallback model" , "language model fallback" ;
    cj:componentOf id:ThisSite ;
    cj:sourceFile "api/ask.js" , "lib/llm.js" , "lib/verify.js" ;
    cj:implements id:RetrievalAugmentedGeneration ;
    cj:uses id:NodeJS ;
    cj:description "A verified fallback to a hosted language model for the questions the question engine is unsure of: the model sees only retrieved evidence, and its answer is checked against it before anyone reads it. It is switched off by default, has been run only against a simulated model, and sends nothing anywhere while off." ;
    cj:article "https://ask.chaipat.cc/wiki/SiteFallback"^^xsd:anyURI ;
    cj:implementationDigest "sha256:04c3b8f3b1a337dea49887e537e1f71ff2963280082116a92547f719200bc90c" .