#community-help

Efficiently Indexing Large Number of Documents in Typesense

TLDR Henrique asked for an efficient way to index 30 million documents using typesense. Kishore Nallan suggested checking the import end-point response for success or error messages.

Powered by Struct AI
2
6mo
Solved
Join the chat
Mar 19, 2023 (6 months ago)
Henrique
Photo of md5-c38d061bdd47b3f1ffb4edfe0a3c5886
Henrique
11:45 AM
Hello! I'm testing typesense and I wanted to know if you guys know of an efficient way to index a large amount of documents (30 million) quickly. I'm currently using sidekiq to index documents, I'm having a throughput of 10k documents per second, but I don't know if typesense is indexing correctly or ignoring a large part of the data
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:51 PM
> but I don't know if typesense is indexing correctly or ignoring a large part of the data
You can check the response of the import end-point. Each line should have {"success": true} or false + error message in case of any failures.