Wiki › Technique

Tokenising, stop-word removal and stemming

Technique

No article has been written for this page yet; it is generated from the knowledge graph.
  • Text preprocessing pipeline Mar 2025
    Text preprocessing pipeline is a command-line tool in Python that cleans a CSV of raw text for natural-language processing: Each text goes through four steps: cleaning: lower-casing and removing punctuation and noise; tokenising into words; removing stop words, such as "the" and "and"; stemming, so "running" and "runs" both become "run".

As data

Everything on this page is read from triples in the knowledge graph. These are the 5 it states about Tokenising, stop-word removal and stemming, and 1 more its ontology implies; the console runs SPARQL over the whole graph, here in your browser.

Download
@prefix id: <https://ask.chaipat.cc/id/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix cj: <https://ask.chaipat.cc/ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .

id:TextPreprocessingSteps
    a cj:Technique ;
    rdfs:label "Tokenising, stop-word removal and stemming" ;
    skos:altLabel "stemming" , "tokenising" , "stop words" .
Works 1
Text preprocessing pipeline