no Friction

Data Flow Architecture

The no Friction system utilizes a multi-stage data flow to ensure proper handling of both structured and unstructured data:

1. Data Ingestion: Email and documents are captured by n8n workflows from monitored sources
2. Metadata Extraction: Basic metadata is stored in Airtable as structured records
3. Content Processing: Gemini processes raw content for text extraction and analysis
4. Embedding Generation: Semantic vector embeddings are created from processed content
5. Vector Storage: Embeddings with linked metadata are stored in Pinecone
6. Relationship Management: Bi-directional linking between structured and vector data
7. Unified Access: Web and chatbot interfaces provide seamless access to all data

Key Integration Points

// Example of key integration between systems
const processNewEmail = async (email) => {
  // Extract metadata for structured storage
  const metadata = {
    from: email.from,
    to: email.to,
    subject: email.subject,
    date: email.date,
    hasAttachments: email.attachments.length > 0
  };
  
  // Store in Airtable
  const record = await airtable.create('emails', {
    ...metadata,
    contactId: findContactId(email.from)
  });
  
  // Process content with Gemini
  const processed = await gemini.process(email.body);
  
  // Store vector embedding in Pinecone
  await pinecone.upsert({
    vectors: [{
      id: `email_${record.id}`,
      values: processed.embedding,
      metadata: {
        source_id: record.id,
        entity_type: 'email',
        content_type: 'email_body',
        timestamp: email.date
      }
    }]
  });
  
  return record.id;
};

Email Processing Workflow

1. Email Monitoring: n8n monitors designated email inboxes at regular intervals
2. Metadata Extraction: Captures sender, recipient, time, subject, and other header information
3. Content Extraction: Email body and attachments are processed separately
4. Airtable Storage: Metadata is stored with appropriate entity associations
5. AI Processing: Content sent to Gemini for text extraction, entity recognition, and embedding generation
6. Vector Storage: Processed content with embeddings stored in Pinecone with CRM keys

Google Drive Integration

1. Folder Monitoring: n8n monitors specified Google Drive folders for new or modified files
2. Metadata Capture: Document metadata (filename, type, modified date) is extracted
3. Airtable Entry: Document reference created in structured database
4. Content Processing: Document content processed by Gemini with optimized chunking strategies
5. Embedding Generation: Vector embeddings created for semantic search capabilities
6. Pinecone Storage: Embeddings stored with metadata and links to Airtable records

Pinecone Configuration

• Index Structure:

  One primary index with namespaces for different content types (emails, documents, notes)

• Vector Dimensions:

  768 or 1024 dimensions depending on selected Gemini embedding model

• Distance Metric:

  Cosine similarity for optimal semantic matching

• Metadata Filtering:

  Capabilities for hybrid search combining vector similarity with metadata constraints

// Pinecone index configuration
const indexConfig = {
  name: 'vector-crm',
  dimension: 768,
  metric: 'cosine',
  pods: 1,
  replicas: 1,
  podType: 'p1.x1'
};

// Example query combining vector search with metadata filter
const queryResponse = await pineconeIndex.query({
  queryVector: embeddings,
  topK: 10,
  filter: {
    entity_type: 'contact',
    source_id: { $in: ['contact_123', 'contact_456'] }
  },
  includeMetadata: true
});

Embedding Generation Process

1. Text Preprocessing:

   Cleaning, normalization, and preparation of content for optimal embedding

2. Chunking Strategy:
   • Documents: Split into semantically meaningful sections (paragraphs or topics)
   • Emails: Headers, body, and attachments as separate embeddings
   • Dynamic chunk sizing based on content complexity
3. Embedding Generation:

   Using Gemini API to create high-dimensional vector representations

4. Metadata Attachment:

   Each vector includes comprehensive metadata for filtering and linking

5. Batch Upload:

   Optimized batch processing with error handling and retries