Any tips for improving the time it takes to reload...
# community-help
j
Any tips for improving the time it takes to reload/reindex when a node in a cluster restarts? We have about 75 million documents roughly 1k each and it takes over an hour for a node to be healthy after it restarts. We have a three node cluster with 40GB RAM and 8 CPU for each node. Looking at the CPU and memory usage when the node is coming back up we are no where CPU or memory limits although it does seem like a single could be close to maxed out while the others are mostly idle. Top shows the brpc_wkr process using 75%. We tried breaking up the collection into eight smaller collections and also tried increasing num-documents-parallel-load but neither made a difference. Anything else we should try?
a
Hey @Josh Handley, are you using Typesense Cloud? Some indexing operations are not CPU-bound, so you won't ever see all the CPU being used. You can try bumping up --db-write-buffer-size, which is available in v29. Let me know if you are in the cloud, since then we'll need to set this on our end.
j
Thanks @Alan Martini. We are self-hosting. I'll try increasing --db-write-buffer-size although I didn't think that Typesearch would do DB writes in this scenario. I'm talking about the situation where the documents have already been uploaded to Typesense but one of the nodes goes down and comes back up and rebuilds the indexes. My limited understanding is that Typesense only writes to the DB when uploading documents,
I bumped db-write-buffer-size up to 419430400 and it didn't make a difference in the time it took for the pod to be healthy after restart.
a
@Josh Handley Could you try these parameters then: 1.
--num-collections-parallel-load
— parallel collection loading 2.
--num-documents-parallel-load
— larger document batches 3.
--max-indexing-concurrency
— higher indexing concurrency Btw, to see an improvement in loading by increasing the collections to 8, the num-collections is the parameter you want. Also, you'll want to make sure you are at least at v29 GA, since we have been adding several reindexing improvements.
j
Thanks. I didn't try increasing --num-collections-parallel-load since the default is NUM_CORES*4 which in our case is already 64. Confirmed by the log message
collection_manager.cpp:421] Loading upto 64 collections in parallel, 1000 documents at a time
. I did try increasing --num-documents-parallel-load from the default of 1000 to 20000 and didn't see a difference. I'll try increasing --max-indexing-concurrency and see if that helps.
We are on v29.0
a
To be fair in v29 we already optimize most of it internally, so these parameters no longer make a big difference. You might possibly have reached the cap for indexing speed on your collection as well. From here on, the way will be optimizing your RAM usage, like using
index: false
on fields on don't intend to search or make any operation (facet, sort, etc) and making sure you are using faceting and sorting only on the fields you are actually performing it.
j
I see. I'll take another pass through the schema but I believe we are indexing as little as we can already. Maybe we can come up with a way to make it lighter. Currently we have multiple "name" fields - one per locale i.e. name.fr_fr, name.en_us_..._ - and we index all of them and then query on name.*. Maybe it would be faster to use a bag of words approach - have a single name field that combines all the locale specific names.
👍 1