# Shadow Mode — Silent Observer

**Status:** Ready for deployment  
**Environment:** `ENV=shadow`  
**Safety Level:** HARD-DISABLED SPEAK

## Dependencies

Shadow mode requires Python 3.11+ with these packages:

```
httpx>=0.27.0    # HTTP client for Telegram API
pyyaml>=6.0      # YAML parsing for family context
```

Install all at once:

```bash
cd /home/hoffmann_admin/.openclaw/workspace/services/icarus
pip install -r requirements-shadow.txt
```

Or manually:

```bash
pip install httpx pyyaml
```

**Note:** Standard library modules used (no install needed): `sqlite3`, `json`, `re`, `csv`, `argparse`, `asyncio`, `logging`, `dataclasses`, `datetime`, `pathlib`, `typing`

## What is Shadow Mode?

Shadow mode is a silent observation state where the bot:
- ✅ Reads real messages from the Family Logistics group
- ✅ Runs tripwire detection on every message
- ✅ Extracts entities using LLM when coordination signals are detected
- ✅ Logs everything to an isolated database
- ❌ **NEVER sends messages** (hard-disabled)
- ❌ **Completely invisible to Aundrea**

## Why Shadow Mode?

Testing entity recognition against synthetic "Miller family" data was worthless. Real Hoffmann family messages contain:
- Real names (Matt, Aundrea, Sully, Harper, Maggie)
- Real patterns of coordination
- Real noise (typos, abbreviations, context)
- Real day-of-week references

Shadow mode validates:
1. Tripwire precision/recall (how well we catch coordination signals)
2. Entity recognition (does "Sully" resolve to "Sullivan")
3. Extraction quality (4W: who, what, when, where)
4. False positive rate

## Architecture

```
Telegram Family Logistics Group
    ↓ (reads only)
ShadowBot (SPEAK_ENABLED=False)
    ↓
Tripwire Detection (Tier 1)
    ↓ (if score ≥ 0.4)
LLM Extraction (Tier 2)
    ↓
shadow.db (isolated database)
    ↓
Daily export for review
```

## File Structure

```
services/icarus/
├── core/
│   ├── observer/
│   │   ├── __init__.py           # Module exports
│   │   ├── shadow_database.py    # shadow.db schema & operations
│   │   ├── shadow_bot.py         # Silent bot (SPEAK_HARD_DISABLED)
│   │   ├── shadow_validator.py   # Metrics & export
│   │   ├── shadow_polling.py     # Long-polling loop
│   │   └── shadow_cli.py         # Management CLI
│   ├── config/
│   │   └── shadow.py             # Shadow configuration
│   └── family_context.yaml       # Real Hoffmann family data
├── shadow.env                    # Environment variables
├── run_shadow.py                 # Main runner
└── SHADOW_MODE.md               # This file
```

## Deployment

### Prerequisites

Shadow mode requires:
- Python 3.11+
- `httpx` (HTTP client)
- `PyYAML` (YAML parsing)

Install dependencies (if not already installed):

```bash
pip3 install httpx pyyaml
# Or on Debian/Ubuntu with restricted system packages:
# pip3 install --break-system-packages httpx pyyaml
```

### 1. Pre-flight Checks

```bash
cd /home/hoffmann_admin/.openclaw/workspace/services/icarus

# Verify dependencies
python3 -c "import httpx; import yaml; print('Dependencies OK')"

# Verify environment
cat shadow.env | grep ENV
# Should show: ENV=shadow

# Verify speak is disabled
python3 -c "import sys; sys.path.insert(0, '/home/hoffmann_admin/.openclaw/workspace/services'); from icarus.core.config import shadow; print('SPEAK:', shadow.SPEAK_ENABLED)"
# Should show: SPEAK: False

# Verify PYTHONPATH is set
export PYTHONPATH=/home/hoffmann_admin/.openclaw/workspace/services:$PYTHONPATH
```

### 2. Initialize Database

```bash
# Initialize shadow database
python3 -m icarus.core.observer.shadow_cli init-db

# Verify
cd ~/.icarus/shadow && ls -la
# Should see: shadow.db
```

### 3. Start Shadow Mode

```bash
# Load environment and run
export ENV=shadow
export PYTHONPATH=/home/hoffmann_admin/.openclaw/workspace/services:$PYTHONPATH
source shadow.env
python3 run_shadow.py
```

The bot will:
- Connect to Telegram
- Start observing messages
- Log to shadow.db
- **Never respond**

### 4. Verify Silent Operation

Send a test message in Family Logistics:
- Bot should NOT respond
- Check logs: `~/.icarus/shadow/logs/`
- Check database: `python3 -m icarus.core.observer.shadow_cli status`

## Daily Operations

### Check Status

```bash
python3 -m icarus.core.observer.shadow_cli status
```

### Export Daily Data

```bash
# Export today's data
python3 -m icarus.core.observer.shadow_cli export

# Export specific date
python3 -m icarus.core.observer.shadow_cli export --date 2026-05-01

# Exports go to: ~/.icarus/shadow/exports/
```

### Review Queue

```bash
# Show extractions awaiting validation
python3 -m icarus.core.observer.shadow_cli review-queue
```

### Manual Validation

```bash
# Mark extraction as validated
python3 -m icarus.core.observer.shadow_cli validate 123 validated \
    --ground-truth-type coordination \
    --ground-truth-who "Matt,Aundrea"

# Mark as rejected
python3 -m icarus.core.observer.shadow_cli validate 124 rejected \
    --notes "Not a coordination message"
```

### Calculate Metrics

```bash
# After validating extractions
python3 -m icarus.core.observer.shadow_cli metrics
```

### Weekly Report

```bash
python3 -m icarus.core.observer.shadow_cli weekly-report
```

## Safety Guarantees

### 1. Hard-Disabled Speak

In `core/observer/shadow_bot.py`:
```python
SPEAK_ENABLED = False  # Compile-time constant

async def _send_message(self, *args, **kwargs):
    logging.error("[ShadowBot] ATTEMPTED TO SEND MESSAGE — BLOCKED")
    return {"ok": False, "error": "SPEAK_DISABLED_IN_SHADOW_MODE"}
```

### 2. Database Isolation

```python
# Shadow database path
DEFAULT_SHADOW_DB_PATH = Path.home() / ".icarus" / "shadow" / "shadow.db"

# Different from:
# - staging: ~/.icarus/staging/icarus.db
# - production: ~/.icarus/icarus.db
```

### 3. Safety Check on Startup

```python
def _safety_check():
    if SPEAK_ENABLED:
        raise RuntimeError("SPEAK_ENABLED must be False in shadow mode")
    if "shadow" not in str(DATA_DIR):
        raise RuntimeError("DATA_DIR must be shadow directory")
```

## Week 0.5 Protocol

### Day 1-7: Silent Observation

1. **Deploy** shadow mode bot
2. **Observe** all Family Logistics messages
3. **Log** to shadow.db
4. **Zero** bot responses (speak disabled)

### Daily (T+1 through T+7)

```bash
# Morning: Check overnight stats
python3 -m icarus.core.observer.shadow_cli status

# Evening: Export and review
python3 -m icarus.core.observer.shadow_cli export
# Review CSV in ~/.icarus/shadow/exports/
```

### End of Week

```bash
# Generate weekly report
python3 -m icarus.core.observer.shadow_cli weekly-report --days 7

# Calculate metrics (requires validations)
python3 -m icarus.core.observer.shadow_cli metrics
```

## Validation Metrics

### Tripwire Performance

- **Precision:** TP / (TP + FP)
  - Of messages that triggered tripwire, how many were actual coordination?
- **Recall:** TP / (TP + FN)
  - Of actual coordination messages, how many triggered tripwire?
- **F1 Score:** 2 * (Precision * Recall) / (Precision + Recall)

### Entity Recognition

- Does "Sully" → "Sullivan"?
- Does "Harper" → "Harper"?
- Does "Matt" → "Matt"?
- Does "me" (from Aundrea) → "Aundrea"?

### False Positive Rate

- How many non-coordination messages triggered tripwire?
- Target: < 10% false positive rate

## Troubleshooting

### ModuleNotFoundError: No module named 'icarus'

**Cause:** Python cannot find the icarus package imports.

**Fix:** Set PYTHONPATH to the services directory before running:
```bash
export PYTHONPATH=/home/hoffmann_admin/.openclaw/workspace/services:$PYTHONPATH
python3 run_shadow.py
```

Or use the shadow.env which now includes PYTHONPATH:
```bash
source shadow.env
python3 run_shadow.py
```

**Root cause:** The icarus package is at `services/icarus/` but imports use `icarus.core...`. 
The PYTHONPATH must point to `services/` (the parent of icarus/) so Python can find the `icarus` package.

### Bot Won't Start

```bash
# Check environment
env | grep ENV
# Should be: ENV=shadow

# Check config loads
python3 -c "import sys; sys.path.insert(0, '.'); from core.config import shadow; print(shadow.SPEAK_ENABLED)"
```

### No Messages Being Logged

1. Check bot is in Family Logistics group
2. Check TELEGRAM_GROUP_ID is correct
3. Check logs for polling errors

### Database Errors

```bash
# Reinitialize database
rm ~/.icarus/shadow/shadow.db
python3 -m icarus.core.observer.shadow_cli init-db
```

## Emergency Shutdown

```bash
# Ctrl+C in terminal running run_shadow.py
# Or:
kill $(pgrep -f run_shadow.py)
```

## Moving to Production

After Week 0.5 validation:

1. Tune tripwire threshold based on metrics
2. Adjust extraction prompts based on false positives
3. Add confirmation buttons for ambiguous extractions
4. Enable SPEAK for confirmed extractions only
5. Migrate to staging with validated configuration

---

**Remember:** The bot is invisible. Aundrea won't know it's there.
That's the point. We're learning from real patterns before we speak.