Handling Large Document Indexing in Typesense

Mauricio asked if OpenAI and Typesense could handle large text document embeddings exceeding OpenAI's limit. Kishore Nallan recommended not to embed large strings due to quality reduction and to handle chunking in application logic as Typesense does not support automatic splitting.

Epi asked about index sizes in relation to document sizes and RAM requirements for their dataset. Kishore Nallan suggested indexing a sample and extrapolating results, and confirmed suitability for indexing large documents like Wikipedia articles in Typesense.

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.