Document Processing Pipeline

Understand how documents are processed, chunked, embedded, and indexed for RAG retrieval in your AI voice agent

Document Processing Pipeline

Learn how your documents are transformed from files and web pages into searchable knowledge for your AI voice agent.

View status: Open your Knowledge Base to see document processing status and chunk counts.

Processing Overview

When you upload a document or crawl a URL, it goes through this pipeline:

┌─────────────────────────────────────────────────────────────────┐
│                    Document Processing Pipeline                  │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Input                                                         │
│   ├── PDF/DOCX Upload ─────┐                                    │
│   └── URL Crawl ───────────┘                                    │
│              │                                                  │
│              ▼                                                  │
│   ┌─────────────────┐                                           │
│   │    Parsing      │  Extract text from source                 │
│   └────────┬────────┘                                           │
│            │                                                    │
│            ▼                                                    │
│   ┌─────────────────┐                                           │
│   │    Chunking     │  Split into ~500 token pieces             │
│   └────────┬────────┘                                           │
│            │                                                    │
│            ▼                                                    │
│   ┌─────────────────┐                                           │
│   │   Embedding     │  Convert to vectors (OpenAI)              │
│   └────────┬────────┘                                           │
│            │                                                    │
│            ▼                                                    │
│   ┌─────────────────┐                                           │
│   │    Indexing     │  Store in Pinecone                        │
│   └─────────────────┘                                           │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Pipeline Stages

1. Parsing

Purpose: Extract raw text from the source.

Source Parser Output
PDF PDF.js / pdfparse Plain text
DOCX mammoth Plain text + structure
TXT/MD Direct read Plain text
HTML (URL) Cheerio Cleaned text

What's extracted:

  • Main content text
  • Headings and structure
  • Lists and tables (as text)

What's ignored:

  • Images (no OCR currently)
  • JavaScript/CSS
  • Navigation elements
  • Headers/footers

2. Chunking

Purpose: Split text into smaller, retrievable pieces.

Parameter Value Description
Chunk Size ~500 tokens Approximate size per chunk
Overlap ~50 tokens Context preserved between chunks
Boundary Sentences Chunks end at sentence boundaries

Why chunk?

  1. LLM context limits: Can't send entire documents to LLM
  2. Precision: Smaller chunks = more targeted retrieval
  3. Relevance: Retrieve only relevant sections

Example chunking:

Original document (1500 words):
┌─────────────────────────────────────────────────┐
│ Section 1: Introduction (300 words)             │
│ Section 2: Features (500 words)                 │
│ Section 3: Pricing (400 words)                  │
│ Section 4: FAQ (300 words)                      │
└─────────────────────────────────────────────────┘

After chunking (4 chunks):
┌────────────────┐  ┌────────────────┐
│ Chunk 1        │  │ Chunk 2        │
│ Intro + start  │  │ Features       │
│ of Features    │  │ continued      │
└────────────────┘  └────────────────┘

┌────────────────┐  ┌────────────────┐
│ Chunk 3        │  │ Chunk 4        │
│ Pricing        │  │ FAQ            │
└────────────────┘  └────────────────┘

3. Embedding

Purpose: Convert text chunks to vectors for semantic search.

Setting Value
Model OpenAI text-embedding-3-small
Dimensions 1536
Batch Size 100 chunks

How it works:

Text: "Our return policy allows returns within 30 days"
        │
        ▼
   OpenAI Embedding API
        │
        ▼
Vector: [0.023, -0.156, 0.089, ..., 0.045]  (1536 dimensions)

Why embeddings?

  • Similar meaning = similar vectors
  • Enables semantic search (understanding, not just keywords)
  • "return policy" matches "refund rules"

4. Indexing

Purpose: Store vectors for fast similarity search.

Setting Value
Database Pinecone
Namespace Per knowledge base
Metadata Document ID, text, source

Stored metadata per chunk:

{
  "documentId": "doc_abc123",
  "documentName": "Return Policy",
  "knowledgeBaseId": "kb_xyz789",
  "chunkIndex": 0,
  "text": "Our return policy allows...",
  "wordCount": 150,
  "sourceType": "file",
  "sourceUrl": null
}

Document Status

Each document displays a status badge:

Status Icon Description Duration
Pending Clock Queued for processing Seconds
Processing Spinner Pipeline in progress Seconds to minutes
Completed Checkmark Ready for retrieval -
Failed X Error occurred -
Stuck Warning Processing timed out (>10 min) -

Status Flow

PENDING ──► PROCESSING ──► COMPLETED
                │
                ├──► FAILED (error)
                │
                └──► STUCK (timeout)

Handling Failed Documents

When a document fails:

  1. View error: Click the info icon to see the error message
  2. Fix issue: Address the cause (file format, content, etc.)
  3. Retry: Click the "Retry" button

Common failures:

  • Invalid file format
  • Empty content
  • Embedding API timeout
  • Network issues

See Troubleshooting for solutions.

Handling Stuck Documents

Documents stuck in "Processing" for over 10 minutes are marked as "Stuck":

  1. Identify: Look for yellow "Stuck" badge
  2. Retry: Click the "Retry" button
  3. Monitor: Watch for successful completion

Why documents get stuck:

  • Server restart during processing
  • Embedding API timeout
  • Database connection lost

Retrieval Process

When your agent needs information during a call:

User: "What's your return policy?"
              │
              ▼
        ┌───────────┐
        │  Embed    │  Convert question to vector
        │  Query    │
        └─────┬─────┘
              │
              ▼
        ┌───────────┐
        │  Vector   │  Find similar chunks in Pinecone
        │  Search   │
        └─────┬─────┘
              │
              ▼
        ┌───────────┐
        │  Rank &   │  Apply minScore filter
        │  Filter   │  Return top K results
        └─────┬─────┘
              │
              ▼
        ┌───────────┐
        │   LLM     │  Generate response using
        │ Response  │  retrieved context
        └───────────┘

Configuration options:

Setting Default Description
topK 5 Number of chunks to retrieve
minScore 0.7 Minimum relevance (0-1)

Learn more: RAG Integration

Processing Time

Typical processing times:

Document Type Size Processing Time
Small PDF <1 MB 5-15 seconds
Large PDF 1-10 MB 30-90 seconds
Single URL - 10-30 seconds
Sitemap (10 pages) - 2-5 minutes
Sitemap (50 pages) - 10-15 minutes

Factors affecting time:

  • File size and complexity
  • Number of pages/chunks
  • Current system load
  • Embedding API response time

Monitoring Processing

Document List View

In your knowledge base, see:

  • Document name and source
  • Processing status
  • Chunk count (when completed)
  • Error messages (when failed)

Chunk Count

The chunk count indicates how many pieces your document was split into:

Chunk Count Document Size
1-5 Small document
5-20 Medium document
20-50 Large document
50+ Very large document

Error Details

For failed documents:

  1. Click the info icon
  2. View full error message
  3. See processing metadata
  4. Check source URL (for crawled content)

Best Practices

For Better Chunking

  • Use clear headings (H1, H2, H3)
  • Write in paragraphs, not walls of text
  • Keep related content together
  • Avoid very long sentences

For Better Retrieval

  • One topic per document
  • Use specific, descriptive titles
  • Include common question phrases
  • Avoid abbreviations without definitions

For Monitoring

  • Check processing status after uploads
  • Retry failed documents promptly
  • Review chunk counts for expected values
  • Test retrieval with sample questions

Next Steps

Questions about processing? Check Troubleshooting or contact support.