https://typesense.org logo
Join Slack
Powered by
A few months ago ( like 8 ) I was trying Typesense
# community-help
m
A few months ago ( like 8 ) I was trying Typesense
j
Could you post in this thread?
m
But I faced a few setbacks with a big amount of data, about 14 million records with a size of 3.6GB
When I used the filters the response time increased from 100-300ms to 2-7 seconds
And what crashed the request, was when I tried to search through the filters
Now I'm wondering if this is still the case, since the other alternative Meilisearch didn't have such a thing
And I had to build my own in the frontend, kinda like a hack which ultimately didn't serve my purpose
j
Interesting, I'm surprised to hear about the crash... But that could happen if there wasn't enough RAM and the OS killed the process.
For eg, here's a larger dataset (32M records, ~10GB on disk) with filters, search as you type, etc: https://songs-search.typesense.org/
In any case, we have made some improvements to performance and memory consumption in the last several months. So I'd recommend giving v0.23.0 a shot
m
Yeah, I mean it wasn't like a crash, the request just timed out after a while (30-40 seconds). I tried with a great server (64 GB of RAM)
j
Did you do a wild-card query (
q=*
) by any chance?
m
Okay, I will try again, although the process for indexing the data is long due to the size of the it so the feedback will come later :D
Yes, that was the problem now that I remember
You told me that haha
😄 1
That's why the data took a long time to load the first time
j
Ah yes, wild card queries are very CPU intensive indeed. We made some minor improvements there, but you'll still see issues with 10s of millions of records
Two new features we added recently to help with this are the ability to turn on server-side caching and server-side search cutoff threshold, so if the search query exceeds your defined threshold, it will return the results found so far
m
Nice, I will definitely try it again
Will report back with feedback soon
👍 1
j
The parameters are called: 
search_cutoff_ms
, 
use_cache
and 
cache_ttl
documented in the tables here: https://typesense.org/docs/0.23.0/api/documents.html#search-parameters
Open in Slack
PreviousNext
Powered by