Search Engine Suite
github.com/mrmcsoftware/searchenginesuite- Category
- Developer Tools
- Rank
- No. 3402Tools index
Previous survey · No. 3275 ·
- Type
- TOOL
- Use case
- Data, Retrieval & Knowledge
- Interfaces
- CLI · Web
- GitHub
- 3 stars
- Date
About
A compact C-based search engine implementing TF-IDF and BM25 ranking with optional proximity search, distributed as indexers, query daemons (with optional OpenSSL security), terminal query clients, and a lightweight Node.js/browser web interface. It avoids external databases, using memory-mapped index files and in-memory daemons for fast lookups, letting users mix and match components for different deployment setups.
What it does
A self-hosted text search engine for your own files, written in C by a single author. You hand it a list of files; it chops each one into overlapping blocks of lines (30 lines, advancing 10 at a time by default), counts the words in every block, and writes one binary index file. Queries rank those blocks with either classic TF-IDF or BM25, the formula most search systems still use for first-pass retrieval. You can query from the terminal, run a small server process that keeps the index loaded, or put a Node.js web page in front of it.
Why it's ranked here
Its value is legibility. The BM25 math sits in one short loop with the textbook constants (k1 of 1.2, length weighting of 0.75), the index format is a documented header followed by plain arrays, and the build needs only a C compiler plus the math library for the core programs. It is Apache 2.0 licensed and ships Linux and Windows build files. Against that, the bundled web layer is not safe to expose to a network, and the repository shows no automated tests. It is a strong teaching and tinkering codebase, not a production search service.
What's good
Almost no dependencies: the indexer and plain query programs are single C files that compile with one command, and only the encrypted daemons need OpenSSL. Results come back as snippets with line numbers rather than whole documents, and overlapping hits from the same file are merged so one passage does not fill the page. The query side supports all-words, exact phrase, literal substring and proximity modes, plus sorting by file name. The README is unusually candid: it labels its own incremental indexers as mostly unnecessary and warns which web backend variant is only useful for understanding.
Tradeoffs
The simplest web backend builds a shell command by pasting the search text between double quotes and running it, so a query containing a quote can execute arbitrary commands on the host. The same server exposes a file viewer that reads any path the browser asks for and echoes back whatever origin the request claims, so it should never face an untrusted network. Word splitting treats only unaccented ASCII letters, digits and underscore as word characters, which fragments non-English text. The README calls the index memory-mapped, but the BM25 daemon reads it into ordinary memory with sequential file reads. The npm test script only prints help text.
How to use it well
Good for a developer who wants to read or modify a complete BM25 pipeline end to end, or who needs quick local search over a code or notes folder on one trusted machine. Build the core programs, write a file list, index it, and query from the terminal or the daemon plus terminal client. If you want the browser interface, keep it bound to localhost and prefer a daemon-backed variant. It does not cover crawling, stemming, Unicode word handling, or vector and semantic search; reach for an established search server when any of those matter.
Technical notes+
src/bm25index.c tokenizes on [A-Za-z0-9_] with lowercasing into a 256-byte buffer, hashes vocabulary with FNV-1a into 131072 buckets, and writes an IndexHeader (magic 0x35324D42, window_size, stride, cdate, 256-byte note) followed by the vocabulary, a float IDF array and per-snippet sparse vectors sorted by token id; defaults are -w30 -s10, input list 'filelist', output bm25.idx. IDF is logf((N - dc + 0.5)/(dc + 0.5)) with no +1, so terms appearing in more than half the snippets get negative weight. src/bm25queryd.c listens on port 8001, loads the index via fread in load_index_file (no mmap in this file), scores with bm25_score using k1=1.2f and beta=0.75f, prefilters with a sorted merge, re-reads the source file for phrase matching, and dedupes hits closer than window_size minus stride lines. src/tfidf.h defines the shared structs and maps uint32_t and logf manually on non-Linux builds. In search.js the /search handler concatenates req.query.q into a string passed to child_process exec and calls JSON.parse on stdout without error handling; /file/ reads an arbitrary absolute path with fs.readFile; every route reflects req.headers.origin into Access-Control-Allow-Origin. src/Makefile.linux links OpenSSL for the *s daemons and clients and Jansson (or the YYJSON shim) for the terminal clients. package.json declares only express ^4.21.2 and a test script that only invokes the daemon-backed web server with -h. src/config.h hard-codes cert.pem and key.pem paths.
Observed
- License
- Apache License 2.0
- Language
- C for indexers, query programs and daemons; Node.js with Express for the web API
- Ranking methods
- TF-IDF and BM25 over overlapping line windows, with separate proximity builds
- Interfaces
- Command-line indexers and query tools, TCP query daemons (plain and OpenSSL), HTTP API and browser UI
- Platform support
- Linux and Windows, with a separate makefile for each and MSVC build instructions in source headers
- Dependencies
- Core C programs need only libc and libm; secure daemons need OpenSSL; terminal clients need Jansson or a YYJSON shim; web layer needs Express
- Web backend query handling
- search.js passes the raw query string into a shell command run with exec
- Tests
- The package.json test script only runs the web server with its help flag
Read from README.md, package.json, LICENSE, src/bm25index.c, src/bm25queryd.c, src/tfidf.h, src/config.h, search.js, src/Makefile.linux.
What it can do
Rank search results using TF-IDF scoring
Text query → Ranked search results
Rank search results using BM25 scoring
Text query → Ranked search results
Perform proximity-based search on indexed documents
Text query → Search results
Build searchable index from documents using indexer component
Documents → Memory-mapped index files
Query the index via a terminal client
Search query → Search results
Query the index via a web interface
Search query → Search results
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.