📄 family-assistant-product-roadmap.md 18,883 bytes Apr 25, 2026 📋 Raw

Project Icarus — Sovereign Context Engine v1.0

Codename: Icarus
From: Daedalus 🎨
To: The Board (Wadsworth / Socrates / Matt)
Date: 2026-04-25
Status: Proposal for discussion

After reviewing the entire family_assistant codebase (~10k lines across 24 modules), here's the path from home project to shipping product.

The Appliance Model

Not a SaaS. A box you plug into Ethernet. Single-tenant appliance running everything locally:

  • Mac mini (M4) with macOS / Python / Ollama / Radicale
  • No cloud dependency, no API costs, no subscriptions
  • Phones sync via standard CalDAV (iOS Settings → Add Account, no app needed)
  • Dashboard served to a kitchen iPad over LAN
  • Telegram for proactive notifications

This eliminates the two biggest blockers flagged in the codebase review:
- LLM cost math? Gone. Customer buys the box. LLM runs locally. No per-email API charges.
- Multi-tenant? Not needed. Each box IS the tenant. No config isolation needed because there's nothing to isolate from.

What Already Works ✅

  • Email → calendar pipeline (IMAP + webhook, with circuit breaker and dedup)
  • LLM appointment extraction (Prompt-as-Code pattern)
  • Recurring event support (deterministic RRULE builder — LLM never generates RRULE directly)
  • Conflict detection + LLM-powered resolution
  • Newsletter parsing with shadow filtering
  • Family Brain / RAG (ChromaDB + nomic-embed-text)
  • Calendar mutation via natural language ("move OT to April 20")
  • Telegram inline buttons (conflict resolve, maintenance done, slot signup)
  • Intent engine (chat-based corrections)

Architecture Changes Needed (~2 weeks)

  1. Remove Cloudflare webhook dependency — IMAP must be primary. Webhook becomes optional Cloudflare optimization.
  2. Default OLLAMA_EMBED_URL to localhost — Currently hardcoded to Gaming PC Tailscale IP.
  3. Add DATA_DIR env var — ChromaDB path uses Path.home() / ".family_assistant", needs configurable.
  4. Add Ollama model auto-download — First-boot script pulls qwen2.5:3b or Llama 3.2 3B.
  5. Bundle Radicale config — Ship a working radicale.conf, zero manual setup.
  6. Write the empty appointment_retry.txt — Currently 0 lines, retry path sends blank prompt to LLM.
  7. Fix ~500 lines of pipeline duplication — IMAP path and webhook path share 80% logic but are copy-pasted.
  8. Remove hermes.py dual notification paths — Direct Telegram API + OpenClaw subprocess = potential double-sends.

Testing Needed (~1-2 weeks)

  • rrule_builder.py unit tests (30 min — deterministic)
  • rejection_engine.py unit tests (30 min — also deterministic)
  • Pipeline integration test: mock email → structured JSON
  • Config module tests: mock env vars
  • GitHub Actions CI runner

Packaging Phase (4-6 weeks)

  1. Docker Compose bundle: family_assistant + Radicale + Ollama + ChromaDB
  2. One-command install script for macOS (Homebrew → Python → models)
  3. Web-based setup wizard (Gmail auth, family members, LAN config)
  4. Dashboard HTMX polish (Daedalus has started this)
  5. Marketing page: "Your family. Your data. Your calendar."

LLM Model for M4 Mac mini

Use Current Appliance Target
Email parsing qwen2.5-coder:7b Llama 3.2 3B or Qwen 2.5 3B
Embeddings nomic-embed-text nomic-embed-text (same, runs fine locally)
Vision/OCR qwen3-vl:8b moondream-2 or Tesseract OCR

7B models work but are overkill for email extraction. 3B models hit 30-40 tok/s on M4 CPU — sub-second per email.

The Hard Part

Not technical. The code is ~80% ready. The hard parts:

  1. Support burden — When someone's Gmail stops syncing, you're the help desk.
  2. Hardware logistics — Pre-configured box or "buy a Mini, run a script?" Pre-configured is a real product, shipping instructions is a hobby.
  3. Radicale on foreign networks — Works great on your network. Different routers, DNS, firewall configs will create edge cases.
  4. LLM quality variance — Prompts that parse Matt's dentist perfectly might struggle with "Your pet has been upgraded to Deluxe Kennel Care Package."

What's Worth Protecting (IP)

  • The LLM prompts (well-crafted, iterated)
  • Shadow filtering / rejection engine pattern
  • Deterministic RRULE builder (prevents LLM hallucination)
  • Prompt-as-Code approach (prompts in text files, not Python)

Everything else is standard Python patterns.

Full codebase review: /home/hoffmann_admin/.openclaw/workspace-daedalus/review/hoffdesk-agents-review.md
DHA Core + Family Assistant review: /home/hoffmann_admin/.openclaw/workspace-daedalus/review/dha-and-family-review.md

Future Modules — Cognitive Load Reduction Roadmap

The appliance's core value is answering family questions before they're asked. Each new module reduces the number of things a parent must remember, hunt for, or text their partner about. Ordered by implementation priority.

Design Principle

Every module must pass the 7 AM test: does this reduce or increase cognitive load for a half-awake parent glancing at a phone or kitchen iPad? If it requires more than two taps or adds visual noise, it doesn't ship.

Tier 1 — Calendar Adjacent (Natural Extensions)

Grocery & Shopping List Management

  • Problem: "Who's picking up milk?" "What did we need from Costco?" is a daily friction point in every household.
  • How: The email pipeline already captures order confirmations and shipping notifications. Extend the rejection engine to detect shopping-related emails and extract line items. Maintain a running shared list accessible via Telegram ("add milk" → list updated) and the kitchen dashboard.
  • LLM use: Extract items from emails ("Your PetSmart order: 2x Purina Pro Plan..."), deduplicate against existing list, categorize (groceries / hardware / pharmacy).
  • Sync: Shared list accessible via dashboard widget and Telegram query ("what's on the list?").
  • Cognitive load removed: No more "who's stopping at the store" texts. No forgetting what you needed.

Package / Delivery Tracking

  • Problem: "When's the package arriving?" is asked daily. Current system creates a one-day calendar event but lacks ETA tracking.
  • How: Extend shipment notification path. Instead of a static calendar event, create a tracker that updates as tracking status progresses. Dashboard shows a "Packages" widget with ETA + status dots.
  • LLM use: Extract tracking numbers + carrier from shipping emails. Poll updates or accept carrier webhook push.
  • Sync: Dashboard widget + Telegram push when status changes ("Your package will arrive tomorrow").
  • Cognitive load removed: No more checking tracking numbers across 4 different carrier websites.

Meal Planning

  • Problem: "What's for dinner?" at 5 PM is existential dread.
  • How: Calendar event type for meals. Recipe suggestion based on what's on the grocery list. "We have chicken and broccoli" → LLM suggests 3 recipes → one tap adds missing ingredients to shopping list.
  • LLM use: Recipe suggestion from available ingredients. Extract prep time. Suggest meals that use overlapping ingredients.
  • Sync: Dashboard shows tonight's dinner. Calendar has the meal as an event. Grocery list auto-updates.
  • Cognitive load removed: No 5 PM "what do I cook" panic. No wasted ingredients.

Tier 2 — Home Awareness (The Brain Expands)

School & Activity Feed Integration

  • Problem: Schools email PDFs, send Remind messages, post to parent portals. These are the least-structured, highest-friction inputs. Missing a field trip permission slip deadline is a real failure mode.
  • How: The document sorter already handles PDFs. Extend to scan school PDFs for field trip dates, early dismissal days, conference slots. Parse Remind-style messages. Create calendar events with the correct child as "owner."
  • LLM use: Multi-page PDF parsing to extract multiple events. "Spring Festival — May 3, 2-4 PM, gymnasium" → structured event per child.
  • Implementation note: Schools use different systems (Remind, ParentSquare, email, paper). The webhook can't catch everything. Create a "forward-to-your-assistant" email address as a universal inbox for all school communications.
  • Cognitive load removed: No need to hunt through 3 different school communication channels to find dates.

Guest / Visitor Awareness

  • Problem: "The plumber is coming Tuesday" creates a calendar event but no context about who, what they need, where to park, or which entrance.
  • How: Extend event creation with guest metadata. The Brain stores: "Plumber John — arriving 9-12 Tuesday, needs access to basement water heater." Dashboard shows a "Today's Visitors" card with relevant context.
  • LLM use: Extract visitor context from appointment emails. "John from ABC Plumbing will be at your home between 9-12 to service the water heater."
  • Cognitive load removed: No scrambling to remember who's coming or what they need when the doorbell rings.

Medicine & Health Tracking

  • Problem: Prescription refills, vaccine schedules, allergy seasons, recurring appointments — all manual tracking for most families.
  • How: Calendar events for recurring prescriptions with auto-refill reminders. Vaccine schedule tracking per child. Dashboard shows "Upcoming: Sullivan's annual checkup" 2 weeks out.
  • LLM use: Parse pharmacy emails for prescription ready / refill notices. Extract medication name, dosage, refill-by date.
  • Cognitive load removed: No more "did I refill that prescription?" worry.

Tier 3 — Proactive Intelligence (The Appliance Becomes Helpful)

Pre-Trip Preparation Briefing

  • Current state: Field trip email → calendar event. Better: appliance detects "Sullivan has field trip to Milwaukee Zoo on Friday" and surfaces permission slip status, lunch needed, bus departure time, weather forecast, and what to bring — as a contextual briefing card, not just a calendar block.
  • How: The Brain's hybrid retrieval (Radicale + ChromaDB) already does this for queries. Make it push: when a new event is created with type "field trip" or "appointment," the pipeline checks the original email for prep requirements and stores them as event metadata. Dashboard surfaces them automatically the night before.
  • LLM use: Parse the original email for action items buried in prose. "Please send a sack lunch, water bottle, and permission slip by Wednesday."
  • Cognitive load removed: No frantic morning-of searches for the permission slip email.

Calendar-Derived Location Awareness

  • Problem: "Where's Dad?" "Did Harper get picked up from practice?" — GPS tracking is invasive, calendar is sufficient.
  • How: The location cache module already exists. Calendar events have locations → the Brain deduces "Matt has dentist at 2 PM at 123 Main St" → when asked "where's Matt?" it answers from calendar context. No phone tracking, no privacy concern — just schedule awareness.
  • LLM use: Answer "where is mom right now?" from calendar context. "She has a meeting at the office until 4 PM."
  • Cognitive load removed: No need to text everyone's location — ask the appliance.

Pattern Detection & Anomaly Alerts

  • Current state: Conflict engine catches simultaneous events from the same email batch only.
  • Better state: The appliance learns family rhythms: Harper's violin lesson is always Tuesdays at 4 PM. If a new event lands on Tuesday at 4 PM, flag proactively: "This conflicts with Harper's recurring violin lesson — reassign or reschedule?"
  • How: The Brain's ChromaDB stores historical events. Add a pattern-detection module that looks at past weekly/monthly events and learns family cadences. New event clashes with learned pattern → alert.
  • LLM use: Analyze historical event clusters to infer recurring patterns not explicitly marked as RRULE.
  • Risk: Pattern detection is hard to get right without false positives. v2 feature — ship after core pipeline is stable.
  • Cognitive load removed: Catches scheduling conflicts before they happen, not after.

Tier 4 — The Ambient Dashboard

The kitchen iPad dashboard is the family's at-a-glance hub. It should surface (at minimum):

  1. Today's timeline — calendar events in chronological order, color-coded by person
  2. Weather + school delays — hyperlocal, checked every morning
  3. Packages arriving today — with ETA windows and carrier
  4. Grocery list — editable, shared across devices, auto-populated from email
  5. Tonight's dinner — from meal plan or "what's in the fridge" suggestion
  6. Today's visitors — who's coming, when, why
  7. Recently processed emails — "2 new events added, 3 skipped" with expand details
  8. System health — "All services running" or actionable alert

Design constraint: Zero clicks to the most important info. Glanceable in under 2 seconds. This is what Daedalus has started building and the HTMX foundation is correct for it.

What NOT to Build

Feature Reason to Skip
Voice control Doubles support burden. Telegram text is sufficient and simpler.
Smart home integration Lights, thermostats, locks are a separate product and a support nightmare.
Photo management Google Photos / iCloud already solve this. Don't compete.
Bill pay Financial liability is too high for a first product. Parse → create reminder, don't touch payment.
AI chat companion Users will expect ChatGPT-level conversation. Keep the appliance's LLM focused on structured extraction and retrieval.
Mobile app CalDAV sync + Telegram + web dashboard covers every device. An app is maintenance debt.

PROJECT ICARUS — Sovereign Context Engine v1.0

1. The Core Thesis: The Cognitive Load Math

The appliance does not sell software; it sells mental bandwidth. Its value is strictly proportional to the number of logistical questions it answers before they are asked — eliminating the need to hunt for emails, send text messages, or hold mental checklists.

Every module earns its place by eliminating a specific information-seeking behavior. If a feature doesn't answer a question that was previously asked through email search, partner-text, or mental recall, it doesn't ship.

2. Core Capabilities & Architectural Requirements

Contextual Briefing Cards

Feature: Transforms standard calendar events into operational briefings — e.g., surfacing permission slips, bus times, lunch needs, weather forecasts, and what-to-bring lists for a school field trip.

Architecture requirement: Must route incoming email attachments (PDFs, images) through the local Vision model to extract data trapped in scanned documents, bypassing the blind spot of text-only webhooks. The document sorter module exists; this extends it from "file to Drive" to "file → vision parse → calendar enrichment."

Calendar-Derived Location Awareness

Feature: Answers "Where is Matt?" or "Where is Aundrea?" using calendar intent rather than invasive GPS tracking.

Architecture requirement: LLM outputs must use probabilistic phrasing — "Matt is scheduled to be at the dentist at 2 PM" — to account for the inevitable drift between calendar plans and physical reality. Never state location as fact.

Temporal Anomaly Detection

Feature: Proactively flags scheduling conflicts against historical family rhythms — e.g., detecting when a new event is booked over Harper's recurring 4 PM Tuesday violin lesson.

Architecture requirement: Requires a parallel SQLite relational database to handle time-series frequency analysis and SQL queries. Vector databases (ChromaDB) are temporally blind and cannot accurately process recurring intervals. The RRULE builder is correct for generating recurrence; this module is for detecting unexpected deviations from learned patterns.

3. The Interface

The Ambient Dashboard

A centralized, glanceable display running on a kitchen iPad. Surfaces: today's chronological timeline, hyperlocal weather, package ETAs, tonight's meal, today's visitors, recently processed events, and system health — all at a glance.

Architecture requirement: Must be built and deployed as a Progressive Web App (PWA) with aggressive WebSocket auto-reconnect logic to defeat iOS's native background-sleep protocols and prevent manual refresh requirements. A dashboard that goes blank after 30 seconds of inactivity on iPad is a dead product.

Push Channels

  • Telegram — primary notification surface. Inline buttons for actions (confirm, reschedule, dismiss).
  • Dashboard (PWA) — ambient glanceable surface. Zero clicks to see today.
  • CalDAV (iOS/Android native calendar) — sync layer. No app needed.

4. Strict Anti-Goals (Out of Scope — V1)

To protect operational stability and minimize support overhead:

  • Voice Control — Text-based Telegram routing only. Voice doubles the attack surface and the support burden.
  • Smart Home Integration — Lights, thermostats, locks are a separate domain. Not this box.
  • Photo Management — Do not compete with Google/Apple Photos. They own this category.
  • Financial Execution — Parse bills to create calendar reminders, but never execute payments. Liability is too high.
  • Mobile App — CalDAV + Telegram + PWA covers every device. A native app is maintenance debt with no marginal value.

The Cognitive Load Math

The appliance reduces cognitive load in direct proportion to the questions it can answer before being asked. Every module eliminates a specific family friction:

Without Appliance With Appliance Module
"Did anyone see the email about Harper's field trip?" Dashboard shows field trip card, prep list, and weather forecast Pre-Trip Briefing
"What day is the dentist again?" Calendar auto-populated from email Pipeline (exists)
"Who's getting milk?" Shared list → one family member grabs it on the way home Grocery List
"When's the package from Amazon arriving?" Dashboard + Telegram push with ETA Package Tracking
"What's for dinner?" Calendar shows planned meal, ingredients on list Meal Planning
"Did I miss the permission slip deadline?" Proactive reminder with link to PDF School Integration

The Brain (RAG) is the most important module that exists. Extending what the Brain knows — packages, meals, visitors, school events — is higher leverage than building new independent features. Each new data source makes the Brain more useful without adding UI complexity.

← Back