It is possible that for some types of traffic / queries , using server side cache is counter productive? I'm wondering what the trade off is for cache usage, if there is a limit to how many queries are cached (for the default 60 second period) etc.

The caching implementation is currently pretty conservation: we cache only 128 query responses and start evicting the oldest inserted element when the cache capacity grows past 128 entries. I can't think of a scenario where this will be counter productive, though. Have you noticed any strange behaviors?

That's very useful info @Kishore Nallan, thank you ! I believe our queries are much too varied to benefit from the current cache implementation with a 128 query response limit since we have over 1000 collections with unique facets for each collection. Definitely explains why we haven't seen a reduction in CPU usage with server side cache 😅 (and Cloudflare doesn't (and can't) cache POST requests used for multi_search queries) 128 does seem very conservative though, perhaps the option to configure it could be exposed in a future build? Wondering if we should set up redis but that feels wrong since Typesense is already in-memory ! I think for CPU usage optimisation, our main concern may be the

max_facet_values

which we've set to 1000 (!?) from the default value of 10 to ensure all facet values are returned due to the dynamic nature of our collections. I would imagine that it's one of the most computationally expensive operations, if not the most expensive one. I'll be making our

max_facet_values

call dynamic which should help there.

Happy to make the number of cache entries configurable. If you had a mapping of which facets are applicable to which collection, then you can reduce the number of facets called for. I wonder if allowing an approximate facet counting mode will also help, but so far nobody has actually asked for that.

Luckily we do have that mapping available easily 🙂 In the process of implementing the change. As far as making the cache entries configurable, I guess it will depend on what the cost is to increase the number. Would going from 128 values to 10000 simply be an overhead of 10k values * the record size?

Each cache entry will be the direct JSON response for that query. So the actual cost for 10K entries is:

size_of_record * num_results_returned * 10,000

Excellent, that's quite reasonable! Since tuning our queries from the front-end, we've seen some great numbers with regard to CPU usage vs queries per second. This cluster now seems over powered 😅 . Previously we regularly were in the 25-30% region under load and it was highly variable (because of unoptimized queries, not Typesense itself)

Oh wow hardly any load now 😂