#community-help

Query on Large Text Document Embedding in OpenAI with Typesense

TLDR Mauricio asked if OpenAI and Typesense could handle large text document embeddings exceeding OpenAI's limit. Kishore Nallan recommended not to embed large strings due to quality reduction and to handle chunking in application logic as Typesense does not support automatic splitting.

Join the chat

Sep 07, 2023 (3 weeks ago)

Photo of md5-1c6a30561de3a358d125fe7bed327f22

Mauricio

02:03 PM

Another quick question. We have some pretty large text documents that we want to avoid breaking up as much as possible. They exceed OpenAI’s 8k token limit so we can’t generate an embedding in a single API call. If we use the automatic embedding generation with the OpenAI embeddings, will typesense error out? We were thinking of chunking the document, generating and embedding for every chunk and then averaging the embeddings to get a final embedding. Could typesense help us out there?

02:04

Mauricio

02:04 PM

I’m pretty sure we could do that ourselves and manually add the embeddings for the large sections but wanted to know if we could avoid that work to better estimate the migration effort

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

02:06 PM

Couple of things:

A) even though openai allows 8K tokens it's not a good idea to embed such large strings because it will reduce the quality of the embeddings.

B) we don't automatically split the text. Any text over the limit is ignored.

Since chunking is domain specific our current thinking is to probably leave it to application logic.

Mauricio

02:10 PM

Gotcha! Thanks

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Finding Similar Documents Using JSON and Embeddings

Manish wants to find similar JSON documents and asks for advice. Jason suggests using Sentence-BERT with vector query and provides guidance on working with OpenAI embeddings and Typesense. They discuss upcoming Typesense features and alternative models.

5mo

Optimum Cluster for 1M Documents with OpenAI Embedding

Denny inquired about the ideal cluster configuration for handling 1M documents with openAI embedding. Jason recommended a specific configuration, explained record size calculation, and clarified embedding generation speed factors and the conditions that trigger openAI.

1mo

Errors in Batch Import with Typesense and OpenAI API

Gustavo encountered errors when importing documents into a collection. After discussion with Jason, it was concluded that the issue stemmed from OpenAI API's handling of batch requests with problematic documents, and improvements to Typesense's error messages and handling were suggested.

3mo