Optimal batch_size for Importing Million Document Dataset
TLDR Ahmad asked the best batch_size
for inserting million data. Jason advised staying at default and using 5000 document batches through API calls.
1
Jan 21, 2022 (22 months ago)
Ahmad
10:30 PMbatch_size
value for inserting a million number of data ? I need a number that will not effect the performance.Jason
10:31 PMAhmad
10:32 PMJason
10:33 PMAhmad
10:37 PM1
Typesense
Indexed 2779 threads (79% resolved)
Similar Threads
Recommendations for Batch Size in /import Endpoint
Dima asked for recommendations on batch size for the /import endpoint. Kishore Nallan suggested a few thousand documents per batch, unless documents are very large.
Optimal batchSize for Typesense Imports/Upserts
Nathan asked about increasing batchSize for imports/upserts. Jason recommended keeping server-side batchSize to default 40, and suggested parallel imports up to the number of CPU cores for optimal performance.
Discussion on Document Inserting Speed and Process
David inquired about document insertion speed, and Jason provided reference values and recommended sending more documents per API call. Both David and Chetan acknowledged the suggestions, with David stating to report back on their experience.