#community-help

Query on Large Text Document Embedding in OpenAI with Typesense

TLDR Mauricio asked if OpenAI and Typesense could handle large text document embeddings exceeding OpenAI's limit. Kishore Nallan recommended not to embed large strings due to quality reduction and to handle chunking in application logic as Typesense does not support automatic splitting.

3mo

Join the chat

Sep 07, 2023 (3 months ago)

Photo of md5-1c6a30561de3a358d125fe7bed327f22

Mauricio

02:03 PM

Another quick question. We have some pretty large text documents that we want to avoid breaking up as much as possible. They exceed OpenAI’s 8k token limit so we can’t generate an embedding in a single API call. If we use the automatic embedding generation with the OpenAI embeddings, will typesense error out? We were thinking of chunking the document, generating and embedding for every chunk and then averaging the embeddings to get a final embedding. Could typesense help us out there?

02:04

Mauricio

02:04 PM

I’m pretty sure we could do that ourselves and manually add the embeddings for the large sections but wanted to know if we could avoid that work to better estimate the migration effort

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

02:06 PM

Couple of things:

A) even though openai allows 8K tokens it's not a good idea to embed such large strings because it will reduce the quality of the embeddings.

B) we don't automatically split the text. Any text over the limit is ignored.

Since chunking is domain specific our current thinking is to probably leave it to application logic.

Mauricio

02:10 PM

Gotcha! Thanks

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Finding Similar Documents Using JSON and Embeddings

Manish wants to find similar JSON documents and asks for advice. Jason suggests using Sentence-BERT with vector query and provides guidance on working with OpenAI embeddings and Typesense. They discuss upcoming Typesense features and alternative models.

7mo

Optimum Cluster for 1M Documents with OpenAI Embedding

Denny inquired about the ideal cluster configuration for handling 1M documents with openAI embedding. Jason recommended a specific configuration, explained record size calculation, and clarified embedding generation speed factors and the conditions that trigger openAI.

3mo

Issues with Embeddings on Collection with 80K Documents

Samuel experienced issues when enabling embeddings on a large collection, leading to an unhealthy cluster. Kishore Nallan suggested rolling back to a previous snapshot, advised on memory calculations for OpenAI embeddings, and confirmed that creating a new cluster should solve the problem.