AEGIS

AEGIS is a browser-native emergency detection system built at the Grayscale Hackathon, where it won 1st place. It uses the live camera, frame-differencing motion analysis, and audio context to classify threats through Claude Vision, then confirms with the user before escalating to a simulated dispatch workflow.

The design bet: real-time safety tooling shouldn't require an app install or a native camera pipeline. Everything runs in the browser over WebRTC and the Web Speech API.

Automated threat detection has two failure modes that matter more than raw accuracy: it misses events, or it cries wolf. A system that escalates on every flicker of motion is worse than useless — it trains people to ignore it.

• Latency — a threat signal is only useful if it's fast, which pushes against sending every frame to a model.
• False positives — escalating without confirmation erodes trust and floods responders.
• Access — native detection stacks are heavy; a browser-first tool works anywhere with a camera.

• Motion-gated vision — a frame-differencing algorithm decides when a scene is worth analyzing, feeding context-aware prompts to Claude Vision for a structured JSON threat assessment instead of streaming every frame.
• Confirmation flow — voice and gesture confirmation via the Web Speech API lets a user verify or cancel before anything escalates, cutting false positives.
• Simulated dispatch — a responder map with geolocation and nearby-service discovery (Leaflet.js) demonstrates the escalation path end to end.
• Provider abstraction — interchangeable vision, audio, and reasoning layers (Claude Vision, YOLOv8, Whisper, Ollama) for on-device and offline fallback modes.

• Cost and latency of vision calls. Sending every frame to a model is slow and expensive. Motion-gating collapsed the call volume to moments that actually changed.
• Structured output from an open-ended model. Threat assessment had to be a reliable JSON contract, not prose — context-aware prompting produced structured, actionable assessments.
• Trust in a hackathon timeframe. The confirmation flow was the difference between a flashy demo and a system whose escalations you'd believe.

• The interesting engineering in a vision system is often when not to call the model, not the model itself.
• A human-in-the-loop confirmation step is a feature, not a compromise — it's what makes an automated alarm trustworthy.
• Next: replayable incident timelines with full decision traceability, and a fully offline on-device YOLOv8 + Whisper path.

After the hackathon I forked AEGIS to my own GitHub and turned the parts I'd proposed into production-minded code, shipped as reviewed pull requests. The core logic is unit-tested; the live browser flow (camera + API key) wasn't run headless, and I note that honestly.

• Provider abstraction layer — the Claude Vision call had been copy-pasted across two files; I extracted it into a single providers/ module behind an analyzeFrame() interface, so the endpoint, model, and parsing live in one place and the backend can be swapped without touching call sites. PR #1 ↗
• Real motion-gating — motion had only shaped the prompt; the system still called the vision model every tick. I added a true gate that skips the call on calm scenes while forcing a periodic heartbeat scan so stationary hazards (fire, smoke) are never missed — cutting token/latency cost without weakening detection. PR #2 ↗
• Test harness + 29 unit tests — Vitest over the core logic: motion math, the gating decision, the provider's JSON parsing, the voice yes/no classifier, and haversine distance — including a documented edge (substring matching means "not" contains "no"). PR #3 ↗