#community-help

Optimal batch_size for Importing Million Document Dataset

TLDR Ahmad asked the best batch_size for inserting million data. Jason advised staying at default and using 5000 document batches through API calls.

22mo

Join the chat

Jan 21, 2022 (22 months ago)

Photo of md5-4a6338a1d6016269c9f234fc1a133144

Ahmad

10:30 PM

Hi Everyone, What should be maximum batch_size value for inserting a million number of data ? I need a number that will not effect the performance.

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

10:31 PM

Ahmad I'd recommend just leaving it at its default value and instead use client-size batching and send say 5000 documents per import API call

Ahmad

10:32 PM

and what if i run multiple api calls inserting 5000 documents each ?

Jason

10:33 PM

Yeah that should be fine. You want to keep an eye on CPU usage to make sure it's not saturated

Ahmad

10:37 PM

its 48%.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Recommendations for Batch Size in /import Endpoint

Dima asked for recommendations on batch size for the /import endpoint. Kishore Nallan suggested a few thousand documents per batch, unless documents are very large.

7mo

Optimal batchSize for Typesense Imports/Upserts

Nathan asked about increasing batchSize for imports/upserts. Jason recommended keeping server-side batchSize to default 40, and suggested parallel imports up to the number of CPU cores for optimal performance.

4mo

Discussion on Document Inserting Speed and Process

David inquired about document insertion speed, and Jason provided reference values and recommended sending more documents per API call. Both David and Chetan acknowledged the suggestions, with David stating to report back on their experience.

2mo