Wiki › Project

pydumper

Project · tool · 2024–2025

pydumper is a set of educational web-scraping scripts, in Go and Python, that collect pages for a search query and store what they find.

Parts

  • A Go crawler that queries the Google Custom Search API and fetches the result pages concurrently with a configurable number of threads. It parses them and saves the results to an SQL database. Its settings (API key, search engine, thread count, result limits, database connection) live in a config.ini.
  • A Python scraper for the same job from the command line, with a shell script that runs several instances at once.
  • Benchmarks and a dictionary of search terms.

The README states that the scripts are for educational purposes only.

See also

In the knowledge graph

pydumper and the 6 things it is linked to. Open any dot for its page, or explore it in the full graph.

Python · skill · Built withPythonGo · skill · Built withGoSQL · skill · Built withSQLSystems from scratch · theme · ThemesSystems from scra…Open source on GitHub · person, role or place · ContextOpen sourceGoogle Custom Search API · service or platform · Works withGoogle Custom Sea… pydumper
6 linked in the graph: 3 skills1 theme1 person, role or place1 service or platform
When
2023202420252026
Dec 2024 – Feb 2025 · 13 commits

As data

Everything on this page is read from triples in the knowledge graph. These are the 21 it states about pydumper, and 2 more its ontology implies; the console runs SPARQL over the whole graph, here in your browser.

Download
@prefix id: <https://ask.chaipat.cc/id/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix cj: <https://ask.chaipat.cc/ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

id:PyDumper
    a cj:Project ;
    rdfs:label "pydumper" ;
    cj:description "Educational dumping scripts in Go and Python." ;
    cj:context id:OpenSource ;
    cj:repository "https://github.com/AppleBoiy/pydumper"^^xsd:anyURI ;
    cj:firstCommit "2024-12-11"^^xsd:date ;
    cj:lastCommit "2025-02-06"^^xsd:date ;
    cj:commitCount 13 ;
    cj:kind "tool" ;
    cj:theme id:ThemeSystems ;
    cj:uses id:Go , id:Python , id:SQL ;
    cj:tier 3 ;
    cj:integrates id:GoogleCustomSearch ;
    cj:article "https://ask.chaipat.cc/wiki/PyDumper"^^xsd:anyURI ;
    cj:activeIn id:Y2024 , id:Y2025 ;
    cj:activity _:b134 , _:b135 , _:b136 .