Shadow Mode — Silent Observer
Status: Ready for deployment
Environment: ENV=shadow
Safety Level: HARD-DISABLED SPEAK
Dependencies
Shadow mode requires Python 3.11+ with these packages:
httpx>=0.27.0 # HTTP client for Telegram API
pyyaml>=6.0 # YAML parsing for family context
Install all at once:
cd /home/hoffmann_admin/.openclaw/workspace/services/icarus
pip install -r requirements-shadow.txt
Or manually:
pip install httpx pyyaml
Note: Standard library modules used (no install needed): sqlite3, json, re, csv, argparse, asyncio, logging, dataclasses, datetime, pathlib, typing
What is Shadow Mode?
Shadow mode is a silent observation state where the bot:
- ✅ Reads real messages from the Family Logistics group
- ✅ Runs tripwire detection on every message
- ✅ Extracts entities using LLM when coordination signals are detected
- ✅ Logs everything to an isolated database
- ❌ NEVER sends messages (hard-disabled)
- ❌ Completely invisible to Aundrea
Why Shadow Mode?
Testing entity recognition against synthetic "Miller family" data was worthless. Real Hoffmann family messages contain:
- Real names (Matt, Aundrea, Sully, Harper, Maggie)
- Real patterns of coordination
- Real noise (typos, abbreviations, context)
- Real day-of-week references
Shadow mode validates:
1. Tripwire precision/recall (how well we catch coordination signals)
2. Entity recognition (does "Sully" resolve to "Sullivan")
3. Extraction quality (4W: who, what, when, where)
4. False positive rate
Architecture
Telegram Family Logistics Group
↓ (reads only)
ShadowBot (SPEAK_ENABLED=False)
↓
Tripwire Detection (Tier 1)
↓ (if score ≥ 0.4)
LLM Extraction (Tier 2)
↓
shadow.db (isolated database)
↓
Daily export for review
File Structure
services/icarus/
├── core/
│ ├── observer/
│ │ ├── __init__.py # Module exports
│ │ ├── shadow_database.py # shadow.db schema & operations
│ │ ├── shadow_bot.py # Silent bot (SPEAK_HARD_DISABLED)
│ │ ├── shadow_validator.py # Metrics & export
│ │ ├── shadow_polling.py # Long-polling loop
│ │ └── shadow_cli.py # Management CLI
│ ├── config/
│ │ └── shadow.py # Shadow configuration
│ └── family_context.yaml # Real Hoffmann family data
├── shadow.env # Environment variables
├── run_shadow.py # Main runner
└── SHADOW_MODE.md # This file
Deployment
Prerequisites
Shadow mode requires:
- Python 3.11+
- httpx (HTTP client)
- PyYAML (YAML parsing)
Install dependencies (if not already installed):
pip3 install httpx pyyaml
# Or on Debian/Ubuntu with restricted system packages:
# pip3 install --break-system-packages httpx pyyaml
1. Pre-flight Checks
cd /home/hoffmann_admin/.openclaw/workspace/services/icarus
# Verify dependencies
python3 -c "import httpx; import yaml; print('Dependencies OK')"
# Verify environment
cat shadow.env | grep ENV
# Should show: ENV=shadow
# Verify speak is disabled
python3 -c "import sys; sys.path.insert(0, '/home/hoffmann_admin/.openclaw/workspace/services'); from icarus.core.config import shadow; print('SPEAK:', shadow.SPEAK_ENABLED)"
# Should show: SPEAK: False
# Verify PYTHONPATH is set
export PYTHONPATH=/home/hoffmann_admin/.openclaw/workspace/services:$PYTHONPATH
2. Initialize Database
# Initialize shadow database
python3 -m icarus.core.observer.shadow_cli init-db
# Verify
cd ~/.icarus/shadow && ls -la
# Should see: shadow.db
3. Start Shadow Mode
# Load environment and run
export ENV=shadow
export PYTHONPATH=/home/hoffmann_admin/.openclaw/workspace/services:$PYTHONPATH
source shadow.env
python3 run_shadow.py
The bot will:
- Connect to Telegram
- Start observing messages
- Log to shadow.db
- Never respond
4. Verify Silent Operation
Send a test message in Family Logistics:
- Bot should NOT respond
- Check logs: ~/.icarus/shadow/logs/
- Check database: python3 -m icarus.core.observer.shadow_cli status
Daily Operations
Check Status
python3 -m icarus.core.observer.shadow_cli status
Export Daily Data
# Export today's data
python3 -m icarus.core.observer.shadow_cli export
# Export specific date
python3 -m icarus.core.observer.shadow_cli export --date 2026-05-01
# Exports go to: ~/.icarus/shadow/exports/
Review Queue
# Show extractions awaiting validation
python3 -m icarus.core.observer.shadow_cli review-queue
Manual Validation
# Mark extraction as validated
python3 -m icarus.core.observer.shadow_cli validate 123 validated \
--ground-truth-type coordination \
--ground-truth-who "Matt,Aundrea"
# Mark as rejected
python3 -m icarus.core.observer.shadow_cli validate 124 rejected \
--notes "Not a coordination message"
Calculate Metrics
# After validating extractions
python3 -m icarus.core.observer.shadow_cli metrics
Weekly Report
python3 -m icarus.core.observer.shadow_cli weekly-report
Safety Guarantees
1. Hard-Disabled Speak
In core/observer/shadow_bot.py:
SPEAK_ENABLED = False # Compile-time constant
async def _send_message(self, *args, **kwargs):
logging.error("[ShadowBot] ATTEMPTED TO SEND MESSAGE — BLOCKED")
return {"ok": False, "error": "SPEAK_DISABLED_IN_SHADOW_MODE"}
2. Database Isolation
# Shadow database path
DEFAULT_SHADOW_DB_PATH = Path.home() / ".icarus" / "shadow" / "shadow.db"
# Different from:
# - staging: ~/.icarus/staging/icarus.db
# - production: ~/.icarus/icarus.db
3. Safety Check on Startup
def _safety_check():
if SPEAK_ENABLED:
raise RuntimeError("SPEAK_ENABLED must be False in shadow mode")
if "shadow" not in str(DATA_DIR):
raise RuntimeError("DATA_DIR must be shadow directory")
Week 0.5 Protocol
Day 1-7: Silent Observation
- Deploy shadow mode bot
- Observe all Family Logistics messages
- Log to shadow.db
- Zero bot responses (speak disabled)
Daily (T+1 through T+7)
# Morning: Check overnight stats
python3 -m icarus.core.observer.shadow_cli status
# Evening: Export and review
python3 -m icarus.core.observer.shadow_cli export
# Review CSV in ~/.icarus/shadow/exports/
End of Week
# Generate weekly report
python3 -m icarus.core.observer.shadow_cli weekly-report --days 7
# Calculate metrics (requires validations)
python3 -m icarus.core.observer.shadow_cli metrics
Validation Metrics
Tripwire Performance
- Precision: TP / (TP + FP)
- Of messages that triggered tripwire, how many were actual coordination?
- Recall: TP / (TP + FN)
- Of actual coordination messages, how many triggered tripwire?
- F1 Score: 2 * (Precision * Recall) / (Precision + Recall)
Entity Recognition
- Does "Sully" → "Sullivan"?
- Does "Harper" → "Harper"?
- Does "Matt" → "Matt"?
- Does "me" (from Aundrea) → "Aundrea"?
False Positive Rate
- How many non-coordination messages triggered tripwire?
- Target: < 10% false positive rate
Troubleshooting
ModuleNotFoundError: No module named 'icarus'
Cause: Python cannot find the icarus package imports.
Fix: Set PYTHONPATH to the services directory before running:
export PYTHONPATH=/home/hoffmann_admin/.openclaw/workspace/services:$PYTHONPATH
python3 run_shadow.py
Or use the shadow.env which now includes PYTHONPATH:
source shadow.env
python3 run_shadow.py
Root cause: The icarus package is at services/icarus/ but imports use icarus.core....
The PYTHONPATH must point to services/ (the parent of icarus/) so Python can find the icarus package.
Bot Won't Start
# Check environment
env | grep ENV
# Should be: ENV=shadow
# Check config loads
python3 -c "import sys; sys.path.insert(0, '.'); from core.config import shadow; print(shadow.SPEAK_ENABLED)"
No Messages Being Logged
- Check bot is in Family Logistics group
- Check TELEGRAM_GROUP_ID is correct
- Check logs for polling errors
Database Errors
# Reinitialize database
rm ~/.icarus/shadow/shadow.db
python3 -m icarus.core.observer.shadow_cli init-db
Emergency Shutdown
# Ctrl+C in terminal running run_shadow.py
# Or:
kill $(pgrep -f run_shadow.py)
Moving to Production
After Week 0.5 validation:
- Tune tripwire threshold based on metrics
- Adjust extraction prompts based on false positives
- Add confirmation buttons for ambiguous extractions
- Enable SPEAK for confirmed extractions only
- Migrate to staging with validated configuration
Remember: The bot is invisible. Aundrea won't know it's there.
That's the point. We're learning from real patterns before we speak.