Defuddle
github.com/kepano/defuddle- Category
- Developer Tools
- Rank
- No. 460Tools index
- Pricing
- Open Source
- Platform
- web · cli
- Type
- TOOL
- Builder
- @kepano
- GitHub
- 9.4k stars
- Latest release
- 0.19.3
- Date
About
A web content extraction library that removes clutter like ads, sidebars, and navigation to extract clean main content from web pages. Returns cleaned HTML or Markdown with metadata extraction.
What it does
Defuddle turns a page document into a stable reading representation. It chooses a likely content root, applies removal and scoring heuristics, standardizes structures such as headings, footnotes, code, callouts, and math, then retries with gentler settings when the first result looks suspiciously short.
Why it's ranked here
The appeal is breadth with inspectable behavior. Browser, Node.js, and terminal interfaces serve different workflows, while debug output explains the chosen content and removals. Detailed normalization for technical material makes it more useful than a plain text scraper. The explicit work-in-progress warning keeps the verdict measured.
What's good
Its fallback strategy avoids blindly trusting one heuristic pass. Metadata covers authorship, publication details, language, images, site identity, schema.org data, timing, and word count. Users can override content selection and individual cleanup stages. Markdown conversion handles footnotes, task lists, callouts, math, code, images, and difficult tables with specific rules.
Tradeoffs
Heuristic extraction can still choose or remove the wrong material, especially on unusual layouts. Node.js consumers need an external DOM implementation, and that entry requires ES modules. Full math and Markdown capabilities add optional packages. Async extraction may contact a third-party API unless disabled. The project itself warns that development remains unfinished.
How to use it well
Use it for clipping, reading views, ingestion pipelines, or terminal conversions where preserving document structure matters. Start with the core browser bundle, choose the fuller bundles only for richer math or Markdown needs, and enable debug output when a site behaves badly. It does not replace browser automation or a general-purpose crawler.
Technical notes+
package.json defines TypeScript declarations, a Commander-backed CLI binary, and three exports: the core package, defuddle/full, and defuddle/node. src/defuddle.ts implements scored extraction, low-content retries, schema.org fallback, and output sanitization. src/node.ts accepts DOM documents and runs parseAsync(), while marking string and whole-JSDOM inputs as deprecated. src/fetch.ts enforces HTML content types, a 5 MB limit, a 10-second timeout, redirects, proxy variables, charset detection, and user-agent fallback. src/markdown.ts builds conversion rules on Turndown. src/cli.ts accepts URLs, files, or stdin and can emit content, JSON, frontmatter, or one property.
Observed
- License
- MIT
- Primary language
- TypeScript
- Package surface
- Published as the defuddle npm package with core, full, and Node.js exports plus TypeScript declarations.
- Interfaces
- Browser library, Node.js library, and command-line interface.
- Installation
- Install through npm, or run the CLI through npx or a global npm installation.
- Runtime support
- Browser documents and Node.js DOM implementations including linkedom, JSDOM, and happy-dom.
- Testing
- package.json defines Vitest test scripts for the default environment and JSDOM.
Read from README.md, package.json, src/cli.ts, src/index.ts, src/node.ts, src/fetch.ts, src/types.ts, src/utils.ts, src/defuddle.ts, src/markdown.ts, src/metadata.ts, src/constants.ts.
What it can do
Extract main content from web pages
Web page URL or HTML → Clean main content without clutter
Remove ads and sidebars from web pages
Web page with ads and navigation elements → Clean web page content
Convert web content to HTML format
Web page → Cleaned HTML
Convert web content to Markdown format
Web page → Markdown text
Extract metadata from web pages
Web page → Page metadata
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.