#community-help

Discussing Indexing and Embedding Performance in Typesense

TLDR Dima had queries about indexing with embedding in Typesense. Kishore Nallan and Jason provided solutions, including reducing documents sent in an API call and running embeddings on a GPU. They facilitated Dima with the latest RC.

Powered by Struct AI

1

Jul 18, 2023 (4 months ago)
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
08:41 AM
Hi team! I’m currently testing built-in embedding feature from the alpha doc. Do you support indexing with embedding for updating imports, for instance with emplace?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:42 AM
Yes updates should be handled.
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
02:04 PM
Great, it’s working now, I did something wrong on the first attempt 🙃

But I faced with another problem: I ran re-indexing and found that while in-build embedding works on indexing it affects search performance. It is totally fine and expected, so I increased available CPU on the node 2 -> 4 -> 8 -> 16, but it’s still not enough. Can I decrease concurrency of embedding / indexing somehow? For now I’m sending 500 rows in one batch, maybe I should decrease this amount?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
03:30 PM
Yeah reducing the number of documents you send in an import API call, and pacing them out would be the way to reduce indexing concurrency
03:32
Jason
03:32 PM
On a side note, if you have over say 100K documents, I would recommend running the embedding on a GPU, which speeds up embedding generation for local models significantly
03:33
Jason
03:33 PM
GPU support is already in the latest build, but we need to put together docs on how to set it up, since it unfortunately involves setting external dependencies like CUDA (too large to bundle within Typesense)
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
03:39 PM
Got it, thank you!
05:04
Dima
05:04 PM
And one last question — does built-in embedding save into snapshot? Or all content have to be reindexed and embedded after every restart?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:05 PM
Once the embeddings are generated, they are indeed saved into snapshot during the next hourly snapshot in the most recent RC builds (earlier builds had a bug that caused embeddings to be regenerated on each restart)
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
05:28 PM
Ah, maybe it’s the reason. Do you have tar.gz with latest RC at hand or it’s better just wait for a release?
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
06:17 PM
Extracted binary, thank you

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Issues with Cluster Upgrade and Embedding Field

Gustavo had issues upgrading their cluster and their embedding field wasn't being filled. Jason helped to solve the upgrade issue and advised re-indexing the documents to solve the embedding field issue. Both problems were successfully resolved.

8

72
3mo

Issues with Embeddings on Collection with 80K Documents

Samuel experienced issues when enabling embeddings on a large collection, leading to an unhealthy cluster. Kishore Nallan suggested rolling back to a previous snapshot, advised on memory calculations for OpenAI embeddings, and confirmed that creating a new cluster should solve the problem.

1

39
2w

Utilizing Vector Search and Word Embeddings for Comprehensive Search in Typesense

Bill sought clarification on using vector search with multiple word embeddings in Typesense and using them instead of OpenAI's embedding. Kishore Nallan and Jason informed him that their development version 0.25 supports open source embedding models. They also resolved Bill's concerns regarding search performance, language support, and limitations in the search parameters.

11

225
4mo

Implementing Semantic Search with Typesense

Erik sought advice for semantic search implementation in Typesense and raised issues around slow document import and excessive latency. Upon implementing advice from Kishore Nallan to try different models, Erik reported faster times, ultimately deciding to rate-limit imports.

1

17
1mo

Optimum Cluster for 1M Documents with OpenAI Embedding

Denny inquired about the ideal cluster configuration for handling 1M documents with openAI embedding. Jason recommended a specific configuration, explained record size calculation, and clarified embedding generation speed factors and the conditions that trigger openAI.

2

12
3mo