Typesense Server Import Batching Issues
TLDR Alex had issues with import batching in production Typesense server, with failed syncs for batch sizes over 100. Kishore Nallan suggested an upgrade to version 0.22.1 and offered to check logs.
1
Dec 16, 2021 (23 months ago)
Alex
03:30 PMclient.collections('collentionName').documents().import(batchOfItems) function. Our test/staging typesense server can handle batch sizes of 400 easily, but our production typesense fails (sometimes silently and sometimes with timeout error) if the batch sizes is above 100 - even though the timeout is set to 20 seconds. Do you guys have any idea why our Test server (which has minimal cpu and memory) can handle much more? It is pretty annoying as we have 2 millions of items to sync ๐ Thanks in advance
Kishore Nallan
03:48 PMKishore Nallan
03:49 PMAlex
03:50 PMAlex
03:51 PMRight now i can make it work, by using 30s timeout and a batch size of 40. But that will take 110 hours to sync all data.
Kishore Nallan
03:52 PMHowever, in general Typesense tries to give preference to searching vs compared to indexing because latencies in search is far worse than some delay in indexing. Again in 0.22.1 is better on this front and will explicitly return a "lagging" message if you try to write too fast. In which case, you can handle that by backing off.
Kishore Nallan
03:52 PM1
Alex
03:52 PM
Typesense
Indexed 2776 threads (79% resolved)
Similar Threads
Revisiting Typesense for Efficient DB Indexing and Querying
kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.
Resolving Typesense v0.22.0 Import Issues
Anton encountered issues importing documents in batches using Typesense v0.22.0. Kishore Nallan suggested using atomic import and proposed a debug build. After multiple trials, they were able to reproduce and fix the issue. Anton confirmed the solution was working.
Resolving Typesense Documents Import Error
Aljosa experienced an error while using typesense `documents().import()`, related to handling of large document arrays. Jason clarified that batch_size controls server-side batching, not client-side. He advised splitting arrays to address the issue and committed to elaborating its functionality in the docs. Aljosa proposed amending the TypeScript types to accommodate batch_size in the import options.
Importing data to Typesense Cloud with JS client
Zaiste attempted to import data to Typesense Cloud but encountered an error. Kishore Nallan explained the error was due to server side backpressure and suggested a curl approach for the import.