Handling Large Document Indexing in Typesense
TLDR Anish asked about handling large documents in typesense, then found their answer within a linked thread.
Sep 11, 2023 (2 months ago)
Indexed 3011 threads (79% resolved)
Query on Large Text Document Embedding in OpenAI with Typesense
Mauricio asked if OpenAI and Typesense could handle large text document embeddings exceeding OpenAI's limit. Kishore Nallan recommended not to embed large strings due to quality reduction and to handle chunking in application logic as Typesense does not support automatic splitting.
Estimating RAM Requirements for Indexing Documents
Epi asked about index sizes in relation to document sizes and RAM requirements for their dataset. Kishore Nallan suggested indexing a sample and extrapolating results, and confirmed suitability for indexing large documents like Wikipedia articles in Typesense.
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.