#community-help

Efficiently Indexing Large Number of Documents in Typesense

TLDR Henrique asked for an efficient way to index 30 million documents using typesense. Kishore Nallan suggested checking the import end-point response for success or error messages.

6mo

Join the chat

Mar 19, 2023 (6 months ago)

Photo of md5-c38d061bdd47b3f1ffb4edfe0a3c5886

Henrique

11:45 AM

Hello! I'm testing typesense and I wanted to know if you guys know of an efficient way to index a large amount of documents (30 million) quickly. I'm currently using sidekiq to index documents, I'm having a throughput of 10k documents per second, but I don't know if typesense is indexing correctly or ignoring a large part of the data

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

12:51 PM

> but I don't know if typesense is indexing correctly or ignoring a large part of the data
You can check the response of the import end-point. Each line should have {"success": true} or false + error message in case of any failures.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Using Typesense to Index Large Amounts of Data

Rafael wants to use Typesense to index 100M documents currently in MongoDBAtlas. Jason affirmed Typesense can handle it and asked for more details.

14mo

Issues with Typesense Import API Document Indexing

Morgan experienced inconsistent document indexing using the Typesense Import API. Jason clarified that Typesense returns a success: true or false for each record in the batch import call.

20mo

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

8mo