Family Assistant

Sovereign family operations hub — email → calendar + RAG, with LLM-powered parsing, conflict resolution, and conversational retrieval. Zero cloud dependencies.

Family Assistant watches a dedicated email inbox, extracts structured event data using a local LLM (no regex, no brittle date parsers), syncs to a self-hosted Radicale CalDAV calendar, and answers questions about past emails using semantic search. When events conflict, it suggests intelligent resolutions. When someone asks "what do the kids need for the field trip?", it pulls the answer from the emails it's already processed.

Why This Exists

Family calendar management is a solved problem — if you use Cozi, Google Family Calendar, or any of the dozen apps out there. But they all share the same weakness: they parse appointment emails with regex. One formatting change from your vet's office and the parser breaks.

Family Assistant uses Prompt-as-Code: a local LLM reads the email and extracts structured JSON. The prompt is easier to update than a regex library, handles edge cases naturally, and doesn't break when a sender changes their subject line format.

More importantly, Family Assistant is sovereign — it runs on your hardware, stores data on your disk, and uses your domain. No Google account required. No OAuth tokens that expire. No API quotas that get suspended. Your calendar, your email, your data.

Architecture

Email → Cloudflare Email Worker → hook.hoffdesk.com → FastAPI Webhook
                                                              │
                                                              ▼
                                                        LLM Triage ─┬─→ Appointment Parse ──→ Radicale CalDAV + Brain
                                                                    │
                                                                    └─→ Newsletter Parse ──→ Radicale CalDAV + Brain
                                                                              │
                                                                              ▼
                                                                    Conflict Detection
                                                                              │
                                                                              ▼
                                                                    LLM Resolution (split / reassign / reschedule)
                                                                              │
                                                                              ▼
                                                                    Hermes Telegram Alert with Inline Buttons

Group Chat ──→ Intent Router ─┬─→ Calendar Mutation (move/cancel/add/rename/remind)
                              ├─→ Question → Hybrid Query (Calendar + Brain RAG)
                              └─→ Chatter → Silence

Location Resolution: Cache → Fuzzy Match → goplaces → Nominatim → Haversine

Key design decisions:
- Sovereign — no Google dependency. Radicale CalDAV on your hardware, Cloudflare for TLS termination and email routing, local Ollama for LLM inference.
- Push architecture — Cloudflare Email Worker webhook replaces IMAP polling. Instant delivery, no bot-like behavior.
- Prompt-as-Code — prompts live in text files, not Python. Update extraction logic without touching code.
- Hybrid retrieval — Radicale calendar is the source of truth for time/date/location; ChromaDB provides detail context (dress codes, what to bring, signup links).
- Intent guardrail — three-way classification prevents hallucinated RAG queries on chatter ("okay", "thanks").
- Cache-first location resolution — every location string passes through a 5-tier resolver (exact cache → fuzzy match → goplaces → Nominatim → haversine estimate) to prevent LLM hallucinations.
- Day-of-week mismatch detection — LLM extracts the claimed day, pipeline validates against the parsed date, warns if they disagree.
- Self-hosted calendar — Radicale CalDAV with bcrypt auth, Cloudflare Tunnel for iOS, flat-file storage. Zero API limits, zero ban risk.

Features

Email Pipeline

• Webhook delivery — Cloudflare Email Worker posts raw RFC 5322 email to FastAPI webhook. No polling, no IMAP, instant delivery.
• Appointment extraction — LLM parses any email format into structured events
• Newsletter extraction — multi-type (events, reminders, actions, info) with dedup
• Recurring events — weekly, biweekly, monthly (e.g. "2nd Wednesday"), with count or end date
• Shadow filter — rejected items downgraded, not deleted; restore from digest
• Auto-ingestion — all parsed emails/newsletters stored in ChromaDB for RAG retrieval
• HTML-to-text — stdlib-only HTML conversion, no external dependencies
• Day-of-week mismatch — warns when email says "Monday" but the parsed date is Tuesday

Calendar Management

• Radicale CalDAV sync — create, update, cancel, dedup against existing events via caldav+icalendar
• iCalendar RRULE support — full recurring event generation
• Conflict detection — finds overlapping events with 5-min grace period
• LLM conflict resolution — split, reassign, or reschedule with priority rules
• Interactive Telegram buttons — resolve conflicts with a tap
• Shared family calendar — both phones sync via single CalDAV account

Hermes Notification System

• Direct Telegram Bot API — reliable delivery, no CLI timeout issues
• Per-event notifications — single message per calendar event with location + travel time
• Day-of-week warnings — ⚠️ when email day claim doesn't match parsed date
• Inline action buttons — conflict resolution, slot selection, task completion

Conversational Retrieval (Family Brain)

• Hybrid query — synthesizes Calendar (temporal truth) + ChromaDB (detail context)
• Semantic search — nomic-embed-text embeddings via local Ollama
• Retrieval-only — answers questions, never offers actions it can't execute
• 30-day backfill — seed the brain from existing processed emails

Location Intelligence

• 5-tier resolution stack — exact cache → fuzzy prefix match → goplaces/Google Places → Nominatim/OSM → haversine estimate
• Home-biased coordinates — locations resolve with travel time from home
• Fuzzy matching — "Golrusk" → "Golrusk Pet Care Center" (min 4 chars)
• Nominatim fallback — free OpenStreetMap geocoder, no API key needed
• Travel time estimation — straight-line distance × 1.4 road factor ÷ 35 mph

Chat-Driven Intent Engine

• Wake words — "Socrates", "calendar", "schedule" (configurable)
• Reply-to context — replying to a bot message carries event context
• Calendar mutations — move, cancel, add, rename, reject, remind
• Rejection engine — "we don't need that mass, ignore it forever" → persistent rule
• Maintenance sentinel — recurring household task tracking with Done buttons
• Slot handler — follow signup links, extract available time slots, tap to add reminder

Reliability

• Global error handler — pipeline crashes push stack traces to admin DM, no silent deaths
• Idempotent ingestion — ChromaDB uses upsert, Calendar uses dedup
• TTL cleanup — 12-month auto-purge for school year context expiration
• Sovereign backup — daily tar of Radicale + ChromaDB + config → local + remote (Gaming PC via Tailscale SSH)
• Dedup — in-memory 24h TTL on webhook, message-id dedup on email pipeline

Quick Start

Prerequisites

• Python 3.10+
• Ollama running locally (or on another machine on your network)
• Radicale CalDAV server (or any CalDAV server)
• Cloudflare account (for Email Worker + Tunnel)
• Telegram bot token (for notifications and interactive features)

Installation

git clone https://github.com/NightKnight64/family-assistant.git
cd family-assistant
pip install -e .

Setup

# Interactive setup wizard
python -m family_assistant setup

# Or non-interactive (creates template files)
python -m family_assistant setup --non-interactive

This creates:
- .env — API credentials template
- family.yaml — your family members, roles, nicknames

Configure Your Family

Edit family.yaml:

family:
  members:
    - name: Alex
      role: dad
      pronouns: he/him
    - name: Jordan
      role: mom
      pronouns: she/her
    - name: Riley
      role: son
      pronouns: he/him
      nicknames: [Rye]        # LLM normalizes these
      needs_adult: true       # Requires adult to transport
    - name: Morgan
      role: daughter
      pronouns: she/her
      needs_adult: true
    - name: Buddy
      role: dog
      needs_adult: true

location:
  home:
    address: "123 Main St, Anytown, ST 12345"
    lat: 44.5054
    lng: -87.9557

Set Up Radicale CalDAV

pip install radicale

See WEBHOOK_DEPLOY.md for full Radicale + Cloudflare + systemd setup instructions.

Key config:
- Bind to 0.0.0.0:5232 (or 127.0.0.1 if behind Cloudflare Tunnel)
- Use bcrypt htpasswd ($2b$ prefix)
- Permissive rights for private home network

Set Up Email Pipeline

1. Cloudflare Email Worker — deployed as hoffdesk-email, code at scripts/email_worker.js (v3)
2. Email Routing — configure assistant@yourdomain.com → Worker
3. Webhook server — install hoffdesk-webhook.service systemd unit
4. Cloudflare Tunnel — map hook.yourdomain.com → localhost:5000

See WEBHOOK_DEPLOY.md for step-by-step instructions.

Set Up LLM

By default, Family Assistant uses a local Ollama instance:

# Install models
ollama pull qwen2.5-coder:7b      # Extraction + synthesis
ollama pull nomic-embed-text       # Embeddings
ollama pull qwen3-vl:8b            # Vision/OCR (optional, for Document Sorter)

Point to a remote Ollama instance in .env:

LLM_URL=http://your-server:11434/v1/chat/completions
LLM_MODEL=qwen2.5-coder:7b
OLLAMA_EMBED_URL=http://your-server:11434/api/embeddings
VISION_LLM_URL=http://your-server:11434/api/chat  # Optional, for Document Sorter

Set Up Telegram Bot

1. Create a bot via @BotFather
2. Get the bot token and your chat IDs
3. Add to .env: TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID, TELEGRAM_DEV_ID

Usage

Email Pipeline

# Process emails from webhook (called by FastAPI webhook server)
python -m family_assistant process           # process unread emails → parse → calendar + brain

# Legacy: IMAP-based processing (if not using webhook)
python -m family_assistant process --dry-run  # preview without creating events
python -m family_assistant process --no-notify # skip Telegram notifications

# Show upcoming calendar events
python -m family_assistant upcoming --hours 168

# Detect scheduling conflicts
python -m family_assistant conflicts --hours 168

# Generate resolution options for conflicts
python -m family_assistant resolve --hours 168

Conflict Resolution

1. Detect — finds events with overlapping time windows (5-min grace period)
2. Resolve — LLM generates 2-3 options based on family priorities:
   - Split — both events happen, parents divide responsibilities
   - Reassign — same event, different adult takes over
   - Reschedule — removes lower-priority event + creates a rebook reminder
3. Handle — you pick an option, it executes the calendar changes

Priority rules (built into the resolution prompt):
- Children's medical/dental > pet appointments > adult personal
- Kids need adults to transport; dogs need adults to transport
- Suggest splitting before rescheduling when possible

Interactive Conflict Alerts

# Scan for conflicts + send Telegram alert with inline buttons
python -m family_assistant conflict-notify

Intent Engine (Chat-Driven Calendar)

# Direct intent parse + execute
python -m family_assistant intent --message "move OT to 4pm tomorrow"

# Routed from Telegram group reply
python -m family_assistant chat-intent --message "cancel the vet" --reply --quoted "Maggie grooming on Friday"

# Inbound hook (OpenClaw message bridge)
python -m family_assistant inbound-hook --message "Socrates, add soccer practice Tuesdays at 4"

# Reject an event type
python -m family_assistant reject "First Communion Mass" "we don't need this"
python -m family_assistant rejections  # list active rejection rules

Family Brain (RAG Queries)

# Ask a question about past emails/newsletters
python -m family_assistant brain-query --question "what do the kids need for the field trip?"

# Check Brain stats
python -m family_assistant brain-stats

# Backfill Brain from the last 30 days of email
python -m family_assistant brain-backfill --days 30

# Purge documents older than 12 months
python -m family_assistant brain-purge

Location Cache

# Resolve a location (cache → goplaces → Nominatim)
python -m family_assistant location --query "Golrusk Pet Care Center"

# Show cached locations
python -m family_assistant location-stats

Maintenance Sentinel

# Check recurring task status + brief
python -m family_assistant maintenance

# Mark a task complete
python -m family_assistant maint-done --event-summary "Change water filter"

Slot Handler (Signup Links)

# Fetch signup slots from a URL
python -m family_assistant click --url "https://signupgenius.com/..." --context "field trip chaperone"

# Handle slot button tap
python -m family_assistant slot-callback --callback-data 'slot|abc123|2'

Backup

# Manual backup (runs daily at 7 AM via cron)
./backup_hoffdesk.sh

# Skip remote copy (Gaming PC offline)
SKIP_REMOTE=true ./backup_hoffdesk.sh

Project Structure

family_assistant/
├── config.py              — All config from env vars + family.yaml + .env auto-load
├── email_fetcher.py       — IMAP fetch, decode, mark-read (legacy, pre-webhook)
├── email_webhook.py       — FastAPI webhook server for Cloudflare Email Worker
├── appointment_parser.py  — LLM extraction + JSON normalization + recurrence
├── newsletter_parser.py   — Multi-type extraction + Phase 1+2 LLM dedup + recurrence
├── calendar_sync.py       — Radicale CalDAV CRUD, dedup, event matching, recurring events
├── rrule_builder.py       — RRULE construction from LLM recurrence dicts
├── conflict_engine.py     — Detection + resolution + response handler
├── conflict_notify.py     — Telegram inline buttons for conflict resolution + rejection
├── rejection_engine.py    — LLM-powered rejection intent parsing + rule persistence
├── family_brain.py        — ChromaDB RAG: embed, ingest, query, hybrid synthesis
├── location_cache.py      — 5-tier location resolution (cache → fuzzy → goplaces → Nominatim → haversine)
├── hermes.py              — Telegram push notifications (events, conflicts, digest, alerts)
├── pipeline.py            — process_emails() + process_webhook_email() orchestration + error handler
├── intent_engine.py       — Chat intent parsing + calendar execution + hybrid question handler
├── intent_router.py       — Inbound gateway: wake word/reply-to-bot → intent classification
├── inbound_hook.py        — OpenClaw message bridge → intent router
├── maintenance_sentinel.py — Recurring household task tracker + alert dedup
├── clicker.py             — Follow URLs → extract signup slots → Telegram buttons
├── slot_handler.py        — Slot button tap → cache lookup → calendar reminder
├── document_sorter.py     — OCR (qwen3-vl) + LLM classify → Google Drive upload (WIP)
├── url_fetcher.py         — Jina Reader for JS-rendered page content
├── cli.py                 — All CLI commands
├── setup.py               — Setup wizard
├── prompts/               — Externalized prompt text files
│   ├── appointment_extract.txt
│   ├── appointment_retry.txt
│   ├── newsletter_extract.txt
│   ├── newsletter_dedup.txt
│   ├── chat_intent.txt
│   ├── conflict_resolve.txt
│   └── email_classify.txt
├── family.yaml.example    — Template for family configuration
└── tests/
    └── test_qa.py         — 14-case QA suite

scripts/
├── backup_hoffdesk.sh     — Daily backup: Radicale + ChromaDB + secrets → local + remote
├── email_worker.js        — Cloudflare Email Worker (posts to webhook)
├── research_agent/         — Research CLI for deep-dive topics
├── research_cli.py         — Research agent command-line interface
└── systemd/                — Systemd service files
    ├── hoffdesk-webhook.service
    ├── radicale.service
    └── family-assistant-webhook.service

Customizing Prompts

The prompts/ directory contains the LLM instructions. Edit them to:
- Add new appointment types (e.g., "school play rehearsal")
- Change output format (add fields, modify field names)
- Adjust conflict resolution priorities
- Add family-specific context (allergies, recurring schedules, etc.)
- Modify intent classification (add wake words, change chatter rules)

Family member names and nicknames are injected automatically from family.yaml.

Environment Variables

Set these in a .env file in the working directory — Family Assistant auto-loads it.

Deployment

See WEBHOOK_DEPLOY.md for full deployment instructions including:

• Radicale CalDAV server setup (systemd, htpasswd, rights)
• FastAPI webhook server (Cloudflare Tunnel, email routing)
• Cloudflare Email Worker deployment
• iPhone CalDAV account configuration
• Backup script setup (local + remote via Tailscale SSH)

Permanent Model Rules

• ✅ qwen2.5-coder:7b — single model for ALL pipeline tasks (extraction, synthesis, intent)
• ✅ qwen3-vl:8b — vision/OCR only (Document Sorter), keep_alive: 0
• ✅ nomic-embed-text — embeddings only
• ⛔ No home-assist model — ever
• ⛔ No deepseek-r1 models — burn thinking tokens, fail JSON
• ⛔ No qwen3:8b — thinking tokens can't be disabled, 3-13x slower

Roadmap

v1.0 ✅ (Complete)

• [x] Email → Calendar pipeline with LLM extraction
• [x] Cloudflare Email Worker webhook (push architecture)
• [x] Radicale CalDAV migration (Google-free)
• [x] Conflict detection + resolution (split/reassign/reschedule)
• [x] Hermes Telegram notification agent (direct Bot API)
• [x] Day-of-week mismatch detection
• [x] Location intelligence (5-tier cache + Nominatim fallback + travel time)
• [x] Interactive conflict buttons in Telegram
• [x] Rejection engine (shadow filter + persistent rules)
• [x] Intent engine (chat-driven calendar mutations)
• [x] Recurring event support (weekly, biweekly, monthly patterns)
• [x] Family Brain RAG (ChromaDB + hybrid query)
• [x] Global error handler (admin DM alerts)
• [x] Maintenance sentinel (recurring task tracking)
• [x] Sovereign backup (local + remote via Tailscale SSH)
• [x] Document Sorter (OCR + classify, WIP)

Post-v1.0 (Deferred)

• [ ] Multi-calendar support (work vs. personal)
• [ ] Docker Compose deployment
• [ ] Refactor prompts to YAML for user configurability
• [ ] Provider abstraction (Gmail → Outlook, iCloud)
• [ ] Web UI for conflict resolution
• [ ] Auth failure circuit breaker (3 consecutive → pause + alert)

License

MIT — see LICENSE.