Skip to navigation
GroundX On-Prem

Debugging GroundX On-Prem

Use pipeline logs, resource metrics, and database records to debug a GroundX On-Prem deployment.

Observability

Install a metrics server to monitor CPU and memory use by pod and node:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Alternatively, install a monitoring tool such as Prometheus. Use these metrics to identify failures caused by insufficient resources.

Profiling Ingest flow

Documents pass through these pods during ingestion:

groundx > upload (if the request contains a URL) >
queue > pre-process > layout-api > layout-process > layout-correct >
layout-ocr + layout-inference > layout-map >
layout-save > layout-webhook > pre-process >
summary-client > summary-api > summary-inference > process

The pipeline uses Kafka topics, API requests, and Redis Celery queues to communicate between pods. Kafka topic definitions list the topics. The same flow with its communication steps is:

groundx > [kafka file-upload] >
upload (if the request contains a URL) > [kafka file-update] >
queue > [kafka file-pre-process] >
pre-process > [api request] >
layout-api > [redis-celery-queue process_queue] >
layout-process > [redis-celery-queue correct_queue] >
layout-correct > [redis-celery-queue ocr_queue + layout_queue] >
layout-ocr + layout-inference > [redis-celery-queue map_queue] >
layout-map > [redis-celery-queue save_queue] >
layout-save > [api request] >
layout-webhook > [kafka file-pre-process] >
pre-process > [kafka file-summary] >
summary-client > [api request] >
summary-api > [redis-celery-queue ] >
summary-inference >[kafka file-process] >
process

Start with a documentID. The ingest endpoint returns a processId. Pass it to get_processing_status_by_id to retrieve the document IDs. Trace a document through pod logs and Kafka topics to find where processing stopped. Check resource metrics at that step. Insufficient resources are a common cause of ingestion failures.

Profiling Data

Access the GroundX On-Prem MySQL database with:

kubectl -n eyelevel exec -it mysql-cluster-pxc-db-pxc-0 -- bash
mysql -u DB_USER -pDB_PASSWORD eyelevel

The processor_relationships table records each document’s processing status:

select * from processor_relationships where document_id='e139aa7a-81bb-44cb-8eb8-a4fb172835cf';

processor_id is auto-incremented, so the mapping is not guaranteed. Check the processor definitions before interpreting a status:

select * from processors where processor_id in (3, 4, 8);

Typical mappings are:

  • 3: layout processing. Completion means the file returned to layout-webhook.
  • 4: the mapping step in pre-process. Completion means the file reached summary-client.
  • 8: document rewriting in summary-client.

Use these records to trace a document through the pipeline.

See Also

To compare deployment options, including managing an on-prem instance with GroundX Agent Harness, see Choose Your GroundX Path.

Protected file delivery

For a 503 response or a failed private upload, check:

  • Compatible API, processor, and gateway images.
  • Policy, cache, and database readiness.
  • Read and write access to private storage.
  • A reachable native signing endpoint and trusted TLS certificates.

Verify file delivery with the deployed storage provider and version before claiming support. Reissue expired native links through the local authenticated API. Never make the storage origin public to bypass an error. See File Access and Sharing.