FirstSight
https://github.com/safishamsi/firstsight- Category
- Health & Fitness
- Rank
- No. 1158Tools index
Previous survey · No. 1164 ·
- Pricing
- Open Source
- Type
- APP
- Builder
- safishamsi
- GitHub
- 25 stars
- Latest release
- v1.0.0
- Date
About
Meta Ray-Ban first-aid guidance — contactless heart rate, facial droop detection, and a Gemini voice agent with JRCALC 2022 clinical GraphRAG.
What it does
FirstSight turns live camera and audio streams into an assisted emergency-response session. A Python backend analyzes faces and pulse-related color changes, then supplies observations to a voice agent that retrieves first-aid guidance. Phones, webcams, or supported glasses provide input, while a React dashboard exposes frames, transcripts, signals, and event traces.
Why it's ranked here
This is a compelling end-to-end demonstration because it connects wearable capture, computer vision, spoken guidance, and operator inspection. The implementation includes thoughtful signal handling and several input options. However, missing facial-droop model artifacts, scattered credential setup, and architecture documentation that still describes planned elements weaken its case as a turnkey system.
What's good
The camera contract stays simple: sources send JPEG frames over WebSocket, keeping capture hardware separate from server logic. The heart-rate path tracks individuals, rejects sustained motion, smooths pulse estimates, reports confidence, and distinguishes poor signal from abnormal ranges. The dashboard makes processor output, conversations, session state, and annotated frames inspectable during development.
Tradeoffs
Facial-droop detection cannot run from the repository alone because its trained model, landmark asset, and calibrated threshold are excluded. Heart-rate output needs a full 150-frame buffer, and motion can delay collection further. Mobile setup requires platform-specific secrets and Meta credentials. The browser demo also needs separate Python and Node runtimes plus an external model API key.
How to use it well
Use FirstSight as a development workbench for testing camera-based first-aid assistance across webcams, phones, and Meta glasses. Start with the browser stream, confirm processor behavior through the debug dashboard, then add mobile capture and voice services. It suits prototype and integration work. It does not supply the missing droop artifacts or clinical deployment assurance.
Technical notes+
server/main.py exposes a binary-frame WebSocket and constructs a CPU YOLOR detector plus DeepSort tracker. server/pipeline.py samples detection every tenth frame, maintains 150-frame per-track buffers, rejects sustained optical-flow motion, and builds reduced Gaussian-pyramid regions. server/signal_processor.py uses Butterworth filtering, FFT peak selection, harmonic weighting, confidence estimation, and exponential smoothing. This shown path does not implement the README's stated CHROM/POS ensemble. backend/app/main.py defines a separate FastAPI surface with unrestricted CORS and optional Logfire instrumentation. requirements.txt does not list MediaPipe despite its import in server/detector.py.
Observed
- License
- MIT, stated by the repository README badge.
- Languages and frameworks
- Python FastAPI backends with a React and TypeScript viewer built through Vite.
- Interfaces
- HTTP endpoints and WebSocket frame ingestion, plus browser and native mobile camera clients.
- Supported platforms
- Browser webcam, iPhone, Android, and Meta Ray-Ban glasses.
- Local install surface
- The browser demo requires Python 3.11 through 3.13, Node 18 or newer, and a Gemini API key.
- Model packaging
- Facial-droop weights, the MediaPipe landmark asset, and the calibrated threshold are not included and must be supplied separately.
Read from README.md, Makefile, requirements.txt, docs/CHANGELOG.md, docs/ARCHITECTURE.md, server/main.py, server/tracker.py, server/detector.py, server/pipeline.py, viewer/vite.config.ts, server/signal_processor.py, viewer/src/App.tsx, backend/app/main.py, viewer/src/main.tsx.
What it can do
Measure heart rate contactlessly
Visual data from camera → Heart rate measurement
Detect facial droop
Facial image or video → Facial droop assessment result
Provide voice-based first aid guidance
Spoken questions or emergency descriptions → Spoken first aid instructions
Query clinical protocols using natural language
Medical emergency questions → JRCALC 2022 protocol recommendations
Process first aid queries through augmented retrieval
Medical situation description → Evidence-based clinical guidance
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.