​

Context Memory System — Architecture

​

1. Data Flow — End to End

​

2. Entity Relationship Diagram

​

3. Ingestion Detail

​

Message Flow

1. Bot receives every Slack channel message (not DMs)
2. Fires POST /context/ingest with organizationId, channelId, channelName, platformUserId, messageTs, threadTs, text
3. Message inserted into channel_messages with processedAt = null
4. URLs extracted from message text via regex, cleaned of Slack formatting (<url|display> -> url)
5. Each URL queued as process-url BullMQ job -> Jina Reader API fetches content -> Claude Haiku summarizes -> stored as observation

​

Batch Triggers

​

Extraction Pipeline

1. Fetch unprocessed messages (limit BATCH_SIZE, default 20)
2. Resolve platformUserId -> user names via platform_identities + users tables
3. Group messages by channel, format with timestamps and display names
4. Send to Claude Haiku 4.5 with extraction prompt
5. Parse JSON array of { entityType, entityId, category, observation, confidence }
6. For each observation: generate embedding via OpenAI, insert into context_observations
7. Mark source messages as processed (processedAt = now)
8. Evaluate proactive actions via context-actions.ts rules

​

4. Query Flow — How the Bot Retrieves Context

​

Context String Format

User profile: **Role & Responsibilities**: ... **Current Focus**: ... **Patterns**: ...

Relevant observations:
- [commitment] Committed to delivering the client report by Friday EOD (2026-03-28)
- [blocker] Blocked on design assets from external vendor (2026-03-27)
- [handoff] Took over newsletter formatting from Rohan (2026-03-26)

​

5. Entity Types and Observation Categories

​

Entity Types

​

Observation Categories

​

Confidence Calibration

• 0.9+: Directly stated fact (“I finished the newsletter”)
• 0.7-0.9: Strong inference (“Rohan always does QC after content — likely a handoff pattern”)
• 0.5-0.7: Weak inference (tone-based mood signals)
• Below 0.5: Not stored

​

6. Nightly Summarization

​

Summary Storage

• summaryType = "profile" — one per entity, upserted nightly
• Unique index on (entity_type, entity_id, summary_type) — ensures one profile per entity
• Embedding generated for the summary text itself (enables semantic search on summaries)
• Old summaries are overwritten, not versioned

​

7. Cleanup & Retention

​

8. Backfill Architecture

• Fetches last 1 year of history
• 2-second delay between Slack API calls (rate limit safe)
• Thread replies fetched for threaded messages
• Uses backfill_state table to track progress per channel

​

9. Auto Mood Detection

​

10. API Endpoints

​

11. Environment Variables

​

12. Key Files

​

13. Database Indexes

-- channel_messages
CREATE INDEX channel_messages_org_processed_idx ON channel_messages(organization_id, processed_at);
CREATE UNIQUE INDEX channel_messages_org_channel_ts_idx ON channel_messages(organization_id, channel_id, message_ts);

-- context_observations
CREATE INDEX context_obs_entity_idx ON context_observations(entity_type, entity_id, category);
CREATE INDEX context_obs_org_entity_idx ON context_observations(organization_id, entity_type);
CREATE INDEX context_obs_expires_idx ON context_observations(expires_at);
-- HNSW index on embedding column for cosine similarity search

-- context_summaries
CREATE UNIQUE INDEX context_summaries_entity_type_idx ON context_summaries(entity_type, entity_id, summary_type);
-- HNSW index on embedding column