#community-help

Processing Bulk 'Contacts' in Typesense

TLDR Alan asks for advice on bulk uploading 'contacts'. Kishore Nallan recommends batching, warns against high batch sizes, and agrees to a meeting.

15mo

Join the chat

Oct 03, 2022 (15 months ago)

Photo of md5-6d168f201743aec43607f72d2864612d

Alan

01:43 PM

Good morning Typesense team. Our project will sometimes have bursts of 'contacts' uploaded to our system that can number in the 20k. We can expect at times this could be as high as 30 or 40k in a minute atm (it'll scale as we gain new customers). So, to make this work efficiently with typesense should these be bulk imported 'together' or can typesense efficiently handle say one customer having a bulk import of 1k every 200ms and another 1k every 200ms or so. Anyway, your advice is much appreciated! TY

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

01:45 PM

We recommend batching things up. 10-20K is better than 1K every second. However I think 1K is also not a bad number. You can maybe start with that and then see if you need larger batches later.

Alan

01:46 PM

Actually, it'd be easier to batch more. Any limit to this?

Kishore Nallan

01:48 PM

At very high batch sizes if the write lags significantly Typesense will return a not ready message to searches to prevent stale data from being returned. The exact number depends on size and type of data indexed.

Alan

01:50 PM

Got it. One other question. We have two cloud servers running with you guys, but we're new to typesense. We'd like to make sure we build this right. Are you guys available for a meeting sometime?

Kishore Nallan

02:03 PM

Yes happy to chat. There is a schedule demo link here: https://cloud.typesense.org/

Alan

02:06 PM

Ty, and appreciate the quick responses as usual!

Kishore Nallan

02:06 PM

You're welcome!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

10mo

Typesense Server Import Batching Issues

Alex had issues with import batching in production Typesense server, with failed syncs for batch sizes over 100. Kishore Nallan suggested an upgrade to version 0.22.1 and offered to check logs.

25mo

Improving Record Retrieval Speed from Typesense

Yoshi sought ways to accelerate Typesense record retrieval. Jason advised upgrading to high availability and using the documents/export endpoint. They also noted a high volume of writes consuming significant CPU capacity as a possible performance factor.

3mo

Troubleshooting Indexing Duration in Typesense Import

Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.

15mo

Optimizing Typesense Implementation for Large Collections

Oskar faced performance issues with his document collection in Typesense due to filter additions. Jason suggested trying a newer Typesense build and potentially partitioning the data into country-wise collections. They also discussed reducing network latency with CDN solutions.

11mo