Data Sources Overview

Pillar's co-pilot uses your knowledge base to answer user questions. This section covers how to connect external data sources to keep your content in sync.

How It Works

Your Content              Pillar                     User
┌─────────────┐      ┌──────────────┐      ┌─────────────────┐
│ GCS Bucket  │      │              │      │                 │
│ S3 Bucket   │─────▶│  Knowledge   │─────▶│    Co-pilot     │
│ Website     │      │  Base Sync   │      │  (Chat Panel)   │
│ Help Center │      │              │      │                 │
└─────────────┘      └──────────────┘      └─────────────────┘

You configure a data source in the Pillar dashboard
Pillar syncs your content on a schedule or on-demand
Content is processed and indexed for AI retrieval
Users ask questions and get answers from your knowledge

Supported Sources

Source Type	Description	Setup Guide
Website Crawl	Automatically crawl and index your website or docs	Dashboard
Google Cloud Storage	Sync documents from a GCS bucket	GCS Setup
Amazon S3	Sync documents from an S3 bucket	S3 Setup
Direct Upload	Upload files directly in the dashboard	Dashboard

Sync Behavior

Automatic Sync

By default, Pillar syncs your content:

Immediately when you first connect a source
Daily to pick up new or updated content
On-demand when you trigger a manual sync

Incremental Updates

Pillar uses smart diffing to:

Only process new or changed documents
Remove content that's been deleted from the source
Preserve any manual corrections you've made

Supported File Types

Type	Extensions
Documents	`.md`, `.mdx`, `.txt`, `.pdf`, `.docx`
Web Pages	`.html`, `.htm`
Structured	`.json`, `.yaml`, `.yml`

Best Practices

Organize Your Content

Use clear, descriptive file names
Group related content in folders
Include metadata (title, description) in your documents

Keep Content Fresh

Set up automatic syncs for frequently updated content
Use webhooks to trigger syncs after deployments
Review sync logs to catch any issues

Optimize for AI

Write clear, concise content
Use headings to structure information
Include examples and code snippets where relevant

Next Steps

Google Cloud Storage Setup — Connect a GCS bucket
Amazon S3 Setup — Connect an S3 bucket

Search documentation