Wiki › Project

Text preprocessing pipeline

Project · tool · 2025

Text preprocessing pipeline is a command-line tool in Python that cleans a CSV of raw text for natural-language processing:

code
python client.py raw.csv processed.csv

Each text goes through four steps:

  1. cleaning: lower-casing and removing punctuation and noise;
  2. tokenising into words;
  3. removing stop words, such as "the" and "and";
  4. stemming, so "running" and "runs" both become "run".

Helpers save the processed data and plot the most frequent words. The repository includes tests, a notebook and sample data.

See also

In the knowledge graph

Text preprocessing pipeline and the 7 things it is linked to. Open any dot for its page, or explore it in the full graph.

Intro to machine learning · work · links hereIntro to machine …Python · skill · Built withPythonNatural language processing · skill · Built withNLPTokenising, stop-word removal and stemming · technique · ImplementsTokenising, stop-…Machine learning and data · theme · ThemesMachine learning …Open source on GitHub · person, role or place · ContextOpen sourceCSV · dataset or file format · File formatsCSV Textpreprocessi…
7 linked in the graph: 1 work2 skills1 technique1 theme1 person, role or place1 dataset or file format
When
2023202420252026
Mar 2025 · 3 commits

As data

Everything on this page is read from triples in the knowledge graph. These are the 20 it states about Text preprocessing pipeline, and 3 more its ontology implies; the console runs SPARQL over the whole graph, here in your browser.

Download
@prefix id: <https://ask.chaipat.cc/id/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix cj: <https://ask.chaipat.cc/ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

id:TextPreprocessing
    a cj:Project ;
    rdfs:label "Text preprocessing pipeline" ;
    cj:description "Command-line pipeline that cleans raw text CSVs for NLP." ;
    cj:context id:OpenSource ;
    cj:repository "https://github.com/AppleBoiy/text-preprocessing-pipeline"^^xsd:anyURI ;
    cj:firstCommit "2025-03-04"^^xsd:date ;
    cj:lastCommit "2025-03-07"^^xsd:date ;
    cj:commitCount 3 ;
    cj:kind "tool" ;
    cj:license "MIT" ;
    cj:theme id:ThemeML ;
    cj:uses id:Python , id:NLP ;
    cj:tier 2 ;
    cj:implements id:TextPreprocessingSteps ;
    cj:handles id:CSVFormat ;
    cj:mentions id:IntroML ;
    cj:article "https://ask.chaipat.cc/wiki/TextPreprocessing"^^xsd:anyURI ;
    cj:activeIn id:Y2025 ;
    cj:activity _:b154 .
Works 1
Intro to machine learning