Bulk Indexing Issue with 3 Million Posts
TLDR Digamber is facing a cURL error 28 while bulk indexing 3 million posts. Kishore Nallan suggests checking post body size and possibly encountering a 503 status code.
Mar 09, 2023 (7 months ago)
Digamber
11:49 AMI have 3 million posts, and i’m bulk indexing and during the indexing process i run tino cURL error 28,
Kishore Nallan
11:50 AMDigamber
11:51 AMKishore Nallan
11:52 AMDigamber
11:52 AMDigamber
11:52 AMKishore Nallan
11:53 AMDigamber
11:53 AMKishore Nallan
11:53 AMKishore Nallan
11:54 AMDigamber
11:57 AMAlso - not sure if i’m getting a 503 error - will verify and check
Digamber
12:06 PMKishore Nallan
12:07 PMDigamber
12:07 PMDigamber
12:07 PM
Typesense
Indexed 2779 threads (79% resolved)
Similar Threads
Resolving JSONL File Import Issues in Python
Jon struggles importing a large JSONL file using Python, encountering decode errors and size restrictions. Kishore Nallan instructs to use curl for imports under 10GB, and references an update to the Python client which could more capably handle large imports.
Bulk Import 50MB JSON Files Error - Timeout and Solutions
madhweep encounters an error while bulk importing JSON files. Kishore Nallan provided help, but the issue persists. Jason intervenes and after troubleshooting, they concluded the cluster had run out of memory causing the issue. The problem was resolved by using a cluster with sufficient memory. Daniel also experienced a similar issue, resolved by increasing the timeout.
Issue with Slow Bulk Indexing for 3 Million Documents
Digamber reports slow indexing performance for 3 million documents. Jason asks for the cluster ID and suggests confirming bulk import usage; Digamber will email a JSONL file and curl command for further investigation.
Typesense Server Bulk Import/Upsert Issue Resolved
Adam was confused about the discrepancy between the successful responses and the actual indexed data while working with a custom WP plugin integrating with Typesense. The issue was a bug related to fetching documents in the wrong order, not a Typesense problem.
Typesense Import Issue with HTTP Code 503 Error
Tomas faced errors while importing to typesense, including an HTTP code 503. Jason identified the issue as CPU exhaustion and recommended slowing down writes or upgrading to at least 4vCPU.