- Category
- Developer Tools
- Rank
- No. 1426Tools index
Previous survey · No. 1432 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- supermemoryai
- GitHub
- 209 stars
- Date
About
AST-aware code chunking for RAG — splits source files along real syntax boundaries instead of byte counts, so retrieval keeps functions intact.
What it does
Code Chunk turns source files into structured records for semantic search. It parses each file, identifies named code entities and nesting, then packages nearby code with scope, signatures, imports, sibling information, and source ranges. It can return all records together or yield them incrementally, with configurable size, context, overlap, and import handling.
Why it's ranked here
The library addresses more than splitting alone. It produces embedding-ready text, preserves source locations, handles concurrent batches, and exposes Promise, streaming, Effect, and WASM variants. The repository also contains a retrieval evaluation harness, but no results are supplied here, so its claimed retrieval benefit remains unquantified.
What's good
Context is unusually practical: each result can carry its enclosing scope, defined entities, signatures, nearby siblings, imports, and recoverable parse errors. Oversized entities split at statement boundaries, while small neighbors can merge. Batch jobs isolate errors by file, report progress, and limit concurrency instead of failing the whole collection.
Tradeoffs
Language coverage stops at TypeScript, JavaScript, Python, Rust, Go, and Java. Detection normally depends on file extensions. Streaming yields results incrementally, but callers still provide the complete source string and parsing happens before yielding. WASM consumers must supply the core parser and each required grammar. The documented overlap default also conflicts between the option table and type comments.
How to use it well
Use it when indexing supported-language repositories for code search or RAG, especially when embeddings need scope and signature clues. Reuse one configured chunker across files, use batch concurrency for repositories, and store source ranges beside embeddings. It does not create embeddings, run a vector database, rank search results, or provide a standalone search service.
Technical notes+
packages/code-chunk/src/index.ts exports Promise, async-generator, Effect Stream, batch, formatting, detection, factory, error, and public type surfaces. packages/code-chunk/src/chunk.ts orchestrates language detection, tree-sitter parsing, entity extraction, scope-tree construction, chunking, and recoverable parse-error attachment. packages/code-chunk/src/batch.ts uses Effect streams for bounded concurrency and converts failures into per-file results. packages/code-chunk/src/wasm.ts provides a separately configured WasmChunker, while packages/code-chunk/src/types.ts defines WASM inputs including ArrayBuffer, Uint8Array, Response, WebAssembly.Module, and URL strings. README.md documents an overlap default of 10 lines, but packages/code-chunk/src/types.ts documents 0, so consumers should verify runtime behavior. packages/eval/src/run.ts and packages/eval/src/swebench/score.ts implement retrieval-only comparison and metrics, but the supplied repository text includes no evaluation results.
Observed
- License
- MIT
- Primary language
- TypeScript
- Packaging
- Private Bun workspace monorepo with packages/* workspaces; the library is installed through npm or Bun as code-chunk.
- Library interfaces
- Promise, async-generator, Effect Stream, reusable chunker, concurrent batch, and WASM-configured APIs.
- Supported source languages
- TypeScript, JavaScript, Python, Rust, Go, and Java.
- WASM platform surface
- WASM parser configuration accepts core tree-sitter and per-language grammar binaries, with Cloudflare Workers described as a target.
- Evaluation structure
- Repository text includes a SWE-bench Lite retrieval-only harness comparing basic repository operations with operations plus semantic search.
Read from README.md, package.json, packages/eval/src/run.ts, packages/code-chunk/src/wasm.ts, packages/code-chunk/src/index.ts, packages/code-chunk/src/batch.ts, packages/code-chunk/src/chunk.ts, packages/code-chunk/src/types.ts, packages/code-chunk/src/wasm.d.ts, packages/code-chunk/src/chunker.ts, packages/eval/src/swebench/git.ts, packages/eval/src/swebench/run.ts, packages/eval/src/swebench/score.ts, packages/eval/src/swebench/types.ts, packages/code-chunk/src/scope/tree.ts.
What it can do
Split source code files into chunks based on syntax boundaries
Source code files → Code chunks split at logical syntax points
Parse source code into Abstract Syntax Tree (AST)
Source code files → AST representation of code structure
Preserve complete functions during code chunking
Source code with function definitions → Intact function blocks as separate chunks
Prepare code chunks optimized for RAG retrieval
Source code files → RAG-ready code chunks with preserved semantic boundaries
Avoid splitting code at arbitrary byte count limits
Source code files of any size → Semantically meaningful code chunks regardless of size
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.