Ingest Tutorial

Introduction

Ingest uploads your files to GroundX and processes them for search.

The ingest pipeline has three main steps:

• Detect document elements, visual content and formatting.
• Use vision-language models to convert content into text that LLMs can use.
• Add document context and metadata.

GroundX accepts multiple file formats without requiring you to convert them to plain text.

This guide covers local files, remote files, directories and websites. For document parsing details, see GroundX Ingest for Parsing.

To have an agent submit documents, poll status and search when ready, see Ingest, Search, and Answer With an Agent.

Getting started

You need:

• A destination bucket ID. Create a bucket with buckets.create or the GroundX dashboard.
• The local path or public URL of the file you want to upload.

You can also supply:

• A display name, which can differ from the original file name.
• A supported file type:

bmp, csv, docx, gif, heif, hwp, ico, jpg, json, 
pdf, png, pptx, svg, tiff, tsv, txt, xlsx, webp

processLevel controls processing. The default, full, adds model-based reprocessing to text extraction and chunking, producing X-Ray and enabling workflow extraction. none runs only text extraction/OCR and basic chunking.

Example:

bucket_id = 6830
file_name = "aristotle-rhetoric.pdf";
file_type = "pdf"
upload_path = "documents/Aristotle-rhetoric.pdf";

Ingesting Individual Files

Use ingest to upload individual files to your bucket.

First, create a GroundX client with your API key.

from groundx import Document, GroundX
client = GroundX(
    api_key="YOUR_API_KEY",
)

Upload a document with ingest:

response = client.ingest(
    documents=[
        Document(
            bucket_id=bucket_id,
            file_name=file_name,
            file_path=upload_path,
            file_type=file_type
        )
    ]
)

Set file_path to a local path or public URL.

The response includes processId and status for the ingest batch:

{
    "ingest": {
        "processId": "23e782ac-3829-4833-965d-e77b4e289885",
        "status": "queued"
    }
}

Poll processId with documents.get_processing_status_by_id to check progress.

For rate limits and error handling during ingest, see the error-handling section of the Direct SDK/API Quickstart.

Ingesting Directories

Use the Python SDK’s ingest_directory method to upload a directory to a bucket.

from groundx import GroundX
client = GroundX(
  api_key="YOUR_API_KEY",
)
client.ingest_directory(
  bucket_id=1234,
  path="/path/to/directory",
)

The method finds documents throughout the directory tree, uploads them in batches and polls each batch for progress. It displays a tqdm progress bar.

Ingesting a Website

GroundX can follow links on a public website and ingest its pages. Submit a starting URL with http:// or https:// and your bucket ID. Optional depth and cap values limit link depth and page count:

{
  "websites": [
    {
      "bucketId": 1234,
      "sourceUrl": "https://docs.example.com",
      "depth": 3,
      "cap": 50
    }
  ]
}

The response includes processId and status; poll it like any other ingest batch. On-premises deployments do not support website crawling.

Adding extra search data

GroundX generates search metadata for your files. Add your own searchData to improve ranking and include context, tags or handling notes in search results.

Example:

search_data = {
    "title": "rhetoric",
    "author": "Aristotle",
    "keywords": ["Ethos", "Pathos", "Logos", "Rhetorical Triangle", "Persuasion"]
}

Adding a pre-search filter

searchData influences ranking and accompanies search results. It does not restrict which documents a search can return.

filter stores key-value metadata used to restrict documents before search. Only matching documents enter the search. Apply filters for access rules, tenant isolation or other inclusion requirements. Use searchData for descriptive context.

Example:

filter_data = {
    "department": "finance",
    "security_level": 2,
    "roles": ["director", "executive"]
}
response = client.ingest(
    documents=[
        Document(
            bucket_id=bucket_id,
            file_name=file_name,
            file_path=upload_path,
            file_type=file_type,
            search_data=search_data,
            filter=filter_data
        )
    ]
)

Both filter and searchData can be updated later without re-ingesting the document. For the full filter query syntax used at search time, see Filtering Content.

Final details

Processing time depends on file size. For file and batch limits, see Supported Document Types and Ingest Capacities.

After processing completes, your content is ready for search and use in LLM responses.

Next Steps

• Ready to query what you just ingested? See the Search Tutorial.
• To have an agent ingest documents, poll status and search, see Ingest, Search, and Answer With an Agent.

Protected ingest

File Access and Sharing covers initial bucket and per-file policies, inheritance and private uploads. Before sending a protected local file, use the bucket’s private-upload flow. This applies when protection comes from the account, bucket or file policy, including protected files in public buckets.

Local, remote and crawl items accept an initial fileAccess policy without a revision. Omitted or null policies inherit account or bucket settings. Split files and crawled documents retain the original item’s policy. Compatible deployment and SDK support are required. Compatible SDK helpers use the same bucket upload endpoint for public and protected files. Hosted and on-prem clients use their configured GroundX API and storage without a hosted upload fallback. Upload issuance does not require read protection.