Data Flow

Data Sources & Integrations

These are example integrations from a production deployment. Your system will have different data sources depending on your workflows.

Example Data Pipelines

Source	Method	Script	Frequency
Google Calendar	OAuth2 API	`calendar_reader.py`	Real-time
Gmail	OAuth2 API	`gmail_extractor.py`	Daily
WhatsApp	SQLite DB query	`whatsapp_extractor.py`	Nightly
Apple Health	XML export parse	`health_export_parser.py`	Weekly
Photos Library	SQLite DB query	`photo_triage.py`	Nightly
Claude Code Sessions	JSONL parse	`weekly_data_fetcher.py`	Weekly
Git Activity	GitHub API	`weekly_data_fetcher.py`	Weekly
Apple Notes	JXA (JavaScript for Automation)	`todo_notes_sync.py`	Periodic

Data Flow Pattern


  Raw Sources ──▶ Collectors ──▶ JSONL/Markdown ──▶ Aggregators ──▶ Knowledge Vault ──▶ Agent Consumption
                  (Python         files              (weekly/        (Tracking/         (context for
                   scripts)                           monthly)        Logs/)              Claude)

Example Local Database Locations (macOS)

Database	Path	Access Method
WhatsApp	`[your WhatsApp DB path]`	Read-only SQLite, Core Data epoch timestamps
Photos	`[your Photos Library path]/database/Photos.sqlite`	Read-only with `immutable=1`
Claude Code	`~/.claude/projects/*/`	JSONL files, parse with Python

Credential Requirements per Integration

Integration	Credential Type	Storage
Google Calendar	OAuth2 client + token	Credential vault
Gmail	OAuth2 client + token	Credential vault
GitHub	Personal Access Token	Credential vault
Anthropic	API key	Credential vault
Your integrations	Varies	Credential vault