Hi team, We are observing high result latency due ...
# community-help
p
Hi team, We are observing high result latency due cpu choking on our self-hosted, single node (no HA) typesense instance when performing vector searches. It has 8 cpu cores available. Receiving ~20 queries per second, with an average processing time of 0.5s. Are there some configurations for vector search we need to be aware of? Also, is there a benchmark available for vector search?
f
Hey @Param Gupta , If you're consistently receiving 20 queries / sec, that rounds up to more than 50 million searches per month, which your 8vCPU cluster may not be able to handle, depending on the amount of documents you have in your dataset. Another thing to note is if you're using an external provider for embedding generation, an inbuilt Typesense model or if you're import and generating the embeddings yourselves. Vector search shouldn't be much slower than regular keyword search, but it can lead to performance deficits depending on the alpha parameters of your queries, and the amount of dimensions your models produce. Could you share more info on the things I outlined here?
p
Hey @Fanis Tharropoulos, We're generating the embeddings ourselves and importing them into typesense. Embedding dimensions: 1408 Alpha parameter for our queries is set to: 1.0 Our index size: ~38,000 documents
f
And do only your vector queries take 0.5s to resolve? Was this something that became apparent after a change, or was this always the case?
p
The bottleneck is encountered whenever we increase the "per_page" parameter. We observed a 10x jump in our p90 latency values when increasing per_page from 50 to 250. There is a similar jump in freetext searches on increasing per_page as well, but those are manageable for us. Are 8 cores not enough to handle these vector parameters and result size combo? Are there some parameters we can tune in our current system to start fetching top 250 docs without big latency jumps? Or do we simply need to move into cpu-heavier machines, or maybe go more memory-heavy?