- Category
- AI Tools
- Rank
- No. 973Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- hkuds
- GitHub
- 3.4k stars
- Date
About
HKUDS's RAG system for video — chat with hours of footage, ask questions, and get cited timestamps and clips back.
What it does
VideoRAG breaks long videos into timed segments, transcribes speech, generates visual captions, and embeds the resulting material. It builds a knowledge graph plus vector indexes, then retrieves relevant text and video segments to answer natural-language questions across one or more videos.
Why it's ranked here
The project combines a serious retrieval pipeline with a usable desktop direction. Its published benchmark covers 164 videos and 602 questions across lectures, documentaries, and entertainment. However, the desktop download is still described as forthcoming, so today’s value rests mainly in the source implementation and research workflow.
What's good
The indexing design uses several complementary signals: transcripts, vision-language captions, video embeddings, text chunks, and entity relationships. It saves processed state, skips videos already indexed, caches language-model responses, and supports comparison across multiple videos. The repository also includes reproduction material for a substantial long-video benchmark.
Tradeoffs
Setup is infrastructure-heavy. Users must start a Python backend, launch an Electron frontend, obtain model checkpoints, and configure language-model services. The algorithm loads a local vision-language checkpoint and defaults to OpenAI models, although Azure OpenAI and Ollama adapters exist. Packaged desktop releases are described as coming soon, with Apple Silicon planned before Windows and Linux.
How to use it well
Choose it for research, internal video archives, lecture collections, or documentary analysis where preprocessing cost is acceptable and cross-video retrieval matters. Index stable collections once, retain the working data, and ask focused questions that benefit from transcript, visual, and relationship evidence. It does not yet replace a turnkey hosted video-search service or a broadly available packaged desktop application.
Technical notes+
VideoRAG-algorithm/videorag/videorag.py defines the main VideoRAG dataclass, splits input into 30-second segments by default, runs transcription and captioning, stores JSON, graph, and vector state, and dispatches videorag queries. VideoRAG-algorithm/videorag/_op.py groups segment text into token-bounded chunks and extracts graph entities and relationships. VideoRAG-algorithm/videorag/_llm.py provides OpenAI, Azure OpenAI, and Ollama adapters with caching and retry behavior. Vimo-desktop/electron.vite.config.ts builds separate Electron main, preload, and React renderer targets. Vimo-desktop/python_backend/videorag_api.py exposes a Flask backend and an HTTP-managed ImageBind service. One typing mismatch is visible: VideoRAG-algorithm/videorag/base.py restricts QueryParam.mode to local, global, or naive, while the runtime dispatch expects videorag modes.
Observed
- Primary implementation
- Python algorithm and backend, with a TypeScript and React Electron desktop client.
- Interfaces
- Python library surface, experiment CLI, Flask HTTP backend, and Electron desktop application.
- Model providers
- Adapters are present for OpenAI, Azure OpenAI, and Ollama.
- Install surface
- The documented usable path is source setup with a Python backend and Electron frontend; desktop downloads are described as forthcoming.
- Platform support
- The repository targets macOS, Windows, and Linux, with Apple Silicon planned as the first packaged beta.
- Benchmark structure
- LongerVideos contains 164 videos, 602 questions, and about 134.6 hours across lectures, documentaries, and entertainment.
Read from README.md, Vimo-desktop/postcss.config.js, Vimo-desktop/tailwind.config.js, Vimo-desktop/electron.vite.config.ts, VideoRAG-algorithm/videorag_longervideos.py, VideoRAG-algorithm/videorag/_op.py, VideoRAG-algorithm/videorag/_llm.py, VideoRAG-algorithm/videorag/base.py, VideoRAG-algorithm/videorag/_utils.py, VideoRAG-algorithm/videorag/prompt.py, VideoRAG-algorithm/videorag/__init__.py, VideoRAG-algorithm/videorag/videorag.py, VideoRAG-algorithm/videorag/_splitter.py, Vimo-desktop/python_backend/videorag_api.py.
What it can do
Chat with video content
Hours of video footage and natural language questions → Conversational responses about video content
Answer questions about video footage
Natural language questions and video files → Specific answers with supporting evidence
Extract relevant video clips
User queries and video footage → Specific video clip segments
Generate cited timestamps
User questions and video content → Precise timestamps referencing relevant moments
Process hours of video footage
Long-form video files → Searchable and queryable video content database
Provide evidence-based video responses
User queries about video content → Answers with timestamp citations and video evidence
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.